Meta Turns Idle Server Gaps Into Smarter Ad Rankings Without Extra Hardware

Large inference fleets can exhaust their aggregate capacity while leaving small pockets of CPUs and accelerators idle in specific regions. Meta’s ads engineering team has published a system that pools this stranded capacity and assigns optional ranking work to it without compromising baseline ad serving.

According to Meta’s technical report, the system uses spare compute to rank more ad candidates or route selected requests to a more compute-intensive model. Meta reports that this improves ad relevance and advertiser outcomes without additional capacity orders. The optional work can be withdrawn when primary workloads reclaim the hardware.

How full fleets leave gaps

Unused slots emerge because traffic, hardware delivery, model launches, and resource requirements rarely align across a large fleet. Meta identifies four recurring mismatches:

• Demand timing: Fleets must support peak traffic even though request volume changes by hour, day, and season.
• Delivery timing: Hardware arrives on supply-chain schedules, while model development and deployment follow separate timelines.
• Resource ratios: A region may have spare accelerators but too few supporting CPUs, or the reverse.
• Hardware compatibility: Workloads may support only specific accelerator vendors, generations, or architectures.

These gaps are often too small, brief, or imbalanced for conventional workload placement. Meta combines eligible slots into a shared opportunistic pool while preserving each workload’s hardware requirements. Any task assigned to the pool must tolerate preemption when baseline demand rises.

Two controls fit uneven supply

Meta’s ads pipeline selects a small set of ads from millions of candidates in less than a second. Early stages retrieve and process broad candidate sets, while later stages apply more expensive models to progressively smaller groups. CPUs handle much of the retrieval and feature processing; accelerators run the ranking models.

Rank More Ads (RMA) widens the candidate funnel. The larger set gives later ranking stages more opportunities to identify a relevant ad, but it increases CPU demand for retrieval and feature processing as well as accelerator demand for inference.

Serve Better Models (SBM) keeps the candidate count fixed and sends a larger share of requests to a heavier model. Upstream CPU demand remains comparatively stable, allowing SBM to consume accelerator capacity that RMA cannot use efficiently. A value-aware router prioritizes requests based on predicted engagement.

Fast defenses protect baseline traffic

Opportunistic capacity can disappear within milliseconds when higher-priority traffic reclaims it. A central controller cannot react quickly enough on its own, so Meta combines workload isolation, minute-scale feedback loops, and local load shedding.

RMA protects baseline traffic through logical isolation. A global budget limits how many extra candidates opportunistic replicas may process. SBM uses physical isolation: separate routing targets and dedicated replicas preserve capacity for the baseline model.

The slower feedback layer adjusts how much optional work enters each path. RMA uses a central proportional-integral-derivative controller, commonly called a PID controller, which responds to current, accumulated, and changing error relative to a target. It monitors tail latency, meaning the response times of the slowest requests, and sets a target candidate count that clients apply to new requests.

SBM runs a parallel loop that sets the target share of requests routed to the heavier model. An explore-and-exploit process samples request segments, measures their incremental benefit, and directs more capacity toward segments with higher estimated value.

Because both feedback loops converge over minutes, server-side and client-side load shedding covers sudden capacity loss. These safeguards react within milliseconds and use value signals to remove the lowest-value optional work first, preventing it from consuming resources reserved for baseline requests.

A reusable architecture for inference fleets

Meta’s design offers a general pattern for production inference systems that have preemptible or fragmented capacity:

1. Map the resource shape. Track CPU, accelerator, memory, region, and hardware compatibility instead of treating spare capacity as a single number.
2. Choose disposable work. Assign tasks that can be reduced, rerouted, or dropped without breaking the required serving path.
3. Separate control timescales. Use central feedback for efficient allocation and local shedding for millisecond-scale protection.
4. Allocate by measured value. Estimate the marginal benefit of optional work and remove low-value tasks first when capacity contracts.

Meta’s earlier engineering posts describe adaptive ranking and tail utilization within ads inference. The opportunistic pool broadens that work to shared allocation across regions and hardware classes.

Meta says it is developing an optimization layer that will allocate opportunistic capacity across RMA and SBM according to measured user and advertiser impact. That layer would turn fragmented fleet capacity into a continuously managed resource, with allocation decisions updated as demand, hardware availability, and model value change.