Meta's Andromeda Retrieves 1 Billion Ads per Second on One Server

Meta has rebuilt its ads retrieval pipeline so a single Grace Hopper server can apply targeting rules, score eligible ads, and select the top candidates at billion-ad scale. The new Andromeda system, co-designed with NVIDIA, combines a retrieval mixture-of-experts model, fused GPU kernels, INT8 quantization, and local index matching. Retrieval determines which ads the heavier ranking stage can inspect, so improvements at this stage expand the pool of useful candidates available downstream.

NVIDIA Grace Hopper systems pair a Grace CPU with a Hopper GPU over a high-bandwidth, coherent connection. That topology lets Meta place targeting, scoring, and selection on one server, reducing the data transfers required by earlier deployments with CPU hosts and remote accelerators.

Memory movement set the ceiling

An ads recommendation request usually passes through two model stages. Retrieval scans millions of eligible candidates and keeps the top K, where K is the requested shortlist size. Ranking then spends more compute on those survivors. An ad removed during retrieval never reaches the more accurate ranking model.

Traditional retrieval models kept computation modest because each candidate required embedding vectors to be loaded into the GPU, processed across several kernels, and written back to high-bandwidth memory, or HBM. Those intermediate tensors generated terabytes per second of memory traffic at Meta's scale. CPU hosts also transmitted large candidate lists to remote accelerators, adding a network constraint alongside the memory workload.

Generative creative tools intensified that pressure by helping advertisers produce hundreds of millions of new ads. A larger candidate pool increases memory and network traffic even when the arithmetic required for each ad remains small.

MoE moves upstream

A mixture-of-experts model divides capacity among specialized subnetworks and uses a learned gate to activate the relevant experts for each input. Andromeda replaces its single user tower, the network that converts a person's context into a representation, with several specialized towers. Those experts incorporate sequential ad engagements, cohort representations, language-model-derived semantic features, and engagement patterns across Meta's services.

An attention-based gating network evaluates each candidate against the available user representations and routes it to the most relevant experts. Sparse routing avoids running the full ensemble for every ad. Meta says the resulting retrieval model approaches downstream ranking quality while using approximately 300 times fewer inference FLOPs than its 2024 retrieval model.

FLOPs measure arithmetic operations, while end-to-end performance also depends on memory reads, tensor writes, synchronization, and network transfers. Andromeda's system changes address each of those costs.

Fusion keeps intermediates on chip

Meta and NVIDIA fused the per-candidate sequence into one GPU kernel, a function that executes in parallel across GPU threads. For each user-ad pair, the kernel:

• gathers the required ad embeddings;
• runs the mixture-of-experts interaction; and
• produces the relevance score.

Intermediate values remain in registers and thread-block shared memory, which sit closer to the GPU's compute units than HBM. The fused path avoids repeatedly materializing tensors in HBM between kernel launches. Its reported bandwidth utilization places the gating operation in the same practical regime as dot-product scoring, where data movement determines throughput.

INT8 quantization reduces each model value from 16 bits to 8 bits, cutting storage and transfer requirements per value in half. Meta reports accuracy comparable to FP16 while the smaller representation increases throughput.

The network hop disappears

Earlier versions assigned index matching and top-K selection to CPU hosts and neural scoring to remote GPUs. Index matching evaluates advertiser-defined targeting rules across dozens of feature categories, filtering the pool before model scoring. Transmitting the surviving candidates and their features consumed the network link as throughput rose.

The co-located design sends a compact payload containing user features and a targeting query to one GPU server. That server performs index matching, candidate scoring, and top-K selection locally, then returns only the selected results. The smaller request and response payloads shift the limiting resource back toward GPU compute.

Targeting indexes presented a separate implementation challenge because ads are created, changed, and expired throughout the day. The GPU index must apply complex Boolean rules while accepting real-time updates. Its throughput exceeds neural scoring throughput because matching performs a cheaper filtering operation; complete request processing also includes scoring, selection, data movement, and orchestration.

Production gains need workload context

Meta reports that Andromeda deployments since 2025 increased conversions on Facebook by 2.4%. Conversions are advertiser-selected outcomes such as purchases or registrations. The company attributes the gain to evaluating more candidates within the same latency budget, improving matches while reducing serving cost per candidate.

Comparable evaluation on another stack requires candidate counts, batch sizes, median and tail latency, power draw, update frequency, model-quality metrics, and the exact server configuration. Meta's figures describe capacity under its workloads; those additional measurements determine cost and portability for other systems.

Meta is also porting Andromeda to NVIDIA's newer Grace Blackwell platform and testing different CPU-to-GPU compute ratios. Index maintenance, request handling, and model execution place different demands on each processor, making that ratio a deployment parameter rather than a fixed property of the model.

A practical test for other recommenders

Andromeda widens the retrieval design space by using sparse routing to control arithmetic, kernel fusion to control HBM traffic, and co-location to control network traffic. Recommendation and search teams evaluating a similar architecture should measure:

• bytes transferred per candidate across HBM and the network, alongside FLOPs;
• which intermediate tensors can remain in registers or shared memory;
• the update and consistency costs of moving dynamic filters onto the GPU;
• quality changes under INT8 or lower-precision inference;
• top-K and filtering costs at realistic candidate distributions;
• median and tail latency under production batching; and
• throughput per watt and per dollar for each CPU-to-GPU ratio.