Meta's Prometheus Supercluster Cuts Training Interruptions by 99% Toward a Million GPUs

Meta has published an engineering account of Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio. The first phase connects hundreds of thousands of GPUs across several buildings and trained Meta’s Muse foundation models, including Muse Wonder. Meta expects the completed system to contain roughly one million distributed GPUs.

For Muse Wonder, Meta reports more than 85% goodput during pre-training and more than 90% during reinforcement learning. Goodput measures the share of elapsed time spent making useful training progress after accounting for interruptions and recovery. Mean time between interruptions reached about 47,000 aggregate GPU-hours during training, then rose to 54,000 GPU-hours after firmware updates.

A training job spanning tens of thousands of accelerators can stop when one host fails, an NVLink lane degrades, or a high-bandwidth memory module produces errors. Prometheus combines spare capacity, fault-aware training software, distributed checkpointing, network redundancy, automated diagnostics, and power controls to limit those interruptions.

A million GPUs across several buildings

Prometheus links conventional data centers, AI-optimized buildings, and rapid-deployment structures through miles of fiber. Those facilities operate as one training system, allowing a single workload to use accelerators installed in separate buildings. The completed first phase powered Muse Wonder training.

Meta currently operates two NVIDIA NVL72-based platforms in Prometheus and plans to add a third. An NVL72 domain connects 72 GPUs through a high-bandwidth scale-up fabric, allowing software to treat the domain as one large accelerator.

Catalina uses the Ariel variant of GB200 NVL72. When NVL72 increased local compute, memory, and NVLink bandwidth, Meta raised scale-out network capacity from 400 to 800 Gb/s per GPU to prevent the external network from constraining the larger accelerator domain.

Clemente moves to GB300 NVL72, roughly doubles attention acceleration, and uses ConnectX-8 SuperNICs to provide 800 Gb/s per GPU without a separate networking design. Meta measured a 5% to 10% performance improvement over GB200 NVL72.

San Miguel will use NVIDIA’s Vera Rubin generation. Meta expects it to improve performance per watt by 10% to 20% over GB300 NVL72. The FP4 and FP8 peak-compute figures use different numerical precision and therefore should not be compared directly.

Ethernet carries one job beyond 100 km

Prometheus separates routine service, storage, and control traffic from GPU communication. A dedicated back-end Ethernet fabric carries training collectives, the synchronized exchanges used to combine gradients and other model state across GPUs. Within a building, each fabric can connect as many as 20,736 accelerators through a high-bandwidth, low-latency topology.

Meta’s initial Disaggregated Scheduled Fabric used sub-packet multipath routing and credit-based flow control. Those mechanisms split traffic across paths and prevent senders from overwhelming receiving links during synchronized bursts. As network interface cards improved packet reordering, Meta shifted to a Non-Scheduled Fabric built from 51.2 Tb/s shallow-buffer Ethernet switches with dynamic load balancing. Both designs run on Meta’s FBOSS network software.

The Backend Aggregation network connects those building-scale fabrics through a hub-and-spoke design. It uses Arista 7800 chassis with deep-buffered Jericho3 silicon, supports links from 2 km to more than 100 km, and supplies several petabits per second between region pairs. Runs longer than 10 km use 800G-ZR coherent optics, while shorter links use lower-cost 2x400G FR4 LITE modules.

NVL72 multiplies the failure domain

NVIDIA’s H100 NVL8 architecture grouped eight GPUs in one scale-up domain. NVL72 expands that domain to 72 GPUs, increasing the number of accelerators affected by a domain-level fault ninefold. A failed GPU can make the entire 72-GPU rack unavailable to a synchronized job.

Meta plans capacity around the rack-level failure domain. Training configurations commonly use powers of two, so each NVL72 domain contributes a 64-GPU slice and retains eight GPUs as local spares. When an active accelerator fails, a spare joins the allocation and the job resumes from its latest checkpoint without waiting for a physical repair.

Whole-rack failures require additional capacity. Meta keeps about 2% extra racks available, so a run with 32,768 active GPUs provisions 523 racks containing 37,656 GPUs. That configuration carries approximately 13% more GPUs than the active workload needs, including both in-rack and rack-level reserves.

Two recovery paths

• Parallelism-Aware Fault Tolerance: If a failure remains within one data-parallel replica, PAFT removes that replica while the rest of the cluster continues training at reduced throughput.
• Distributed checkpointing: Larger failures return the job to its latest saved state. Delta checkpoints store only changed data, while in-memory staging reduces the time spent writing checkpoints.

Interruption rates fall by 99%

During Prometheus’s initial phase in mid-2025, Meta recorded nearly 1,000 interruption events per 8,000 GPUs each day. A year of hardware, firmware, networking, and operations work reduced that rate by 99%. The company’s target is two interruptions per 8,000 GPUs per day, equivalent to roughly 100,000 GPU-hours between interruptions.

Failures that emerge at cluster scale

• NVLink signal integrity: More than 600,000 NVLink connections exposed marginal signal degradation. Continuous telemetry, L11 diagnostic burn-in tests, and automated isolation now identify weak links before production use.
• HBM and LPDDR faults: Higher memory capacity increased the number of potential failures. Meta added HBM firmware tuning, memory scrubbing, improved LPDDR error correction, and pattern-based retirement of components likely to fail.
• NVLink fabric health: Millisecond-level monitoring detects degraded switches and links, allowing software to rebalance traffic automatically.
• Back-end network events: Planar network topologies route traffic around failed fibers without pausing synchronized GPU communication.

Prometheus’s fleet-management layer can distribute firmware to more than 100,000 GPUs within a few hours. Meta also reduced the 90th-percentile upgrade time for a single pod from more than seven hours to two hours, meaning nine out of ten pod upgrades now finish within that window.

Power software absorbs megawatt swings

Synchronized training produces rapid changes in electrical demand as tens of thousands of GPUs move between computation and communication. Meta observed cluster load fluctuations of 30% to 40%, while starting or pausing a large job could change demand by tens of megawatts within seconds.

Meta developed Dynamo Dimmer to adjust GPU power limits as the cluster approaches its electrical ceiling, allowing training to continue under a controlled power envelope. Rack-level batteries, capacitor trays, facility-scale battery storage, and utility coordination provide additional protection against abrupt load changes.

Cooling depends on the building type. Rapid-deployment facilities combine liquid cooling with air-assisted systems, while newer AI-optimized buildings use closed-loop, direct-to-chip liquid cooling.

What cluster builders can use

• Capacity planning must follow failure domains. Larger scale-up fabrics require local spares, reserve racks, and schedulers that understand which components can fail together.
• Delivery forecasts need operational metrics. Goodput and mean time between interruptions reveal how much installed compute advances a training run. Peak FLOPS alone does not capture recovery time, checkpoint overhead, or degraded operation.
• Checkpointing belongs in the system design. At this scale, storage bandwidth, in-memory staging, checkpoint cadence, and restart orchestration directly affect training throughput.
• Network telemetry must reach component-level faults. Automated detection and quarantine can keep marginal links, memory modules, and switches from repeatedly interrupting large jobs.
• Power management becomes part of orchestration. Job schedulers and GPU controls must coordinate with rack, facility, and utility limits when synchronized workloads can shift demand by megawatts.

What Meta’s figures leave open

Meta’s results come from internal production measurements rather than an independent benchmark. Comparisons with other clusters require matching workload structure, numerical precision, checkpoint cadence, parallelism strategy, and the definitions used for goodput and interruptions.

The engineering account does not provide total construction cost, operational energy consumption, model-training compute, detailed network oversubscription, or a completion date for the million-GPU configuration. Those omissions limit economic and environmental comparisons, even though the published reliability data offers a detailed view of operating a cluster beyond 100,000 GPUs.

The design moves to Louisiana

Meta plans to apply the Prometheus architecture to Hyperion, its forthcoming supercluster in Richland Parish, Louisiana. Hyperion is being designed with closed-loop liquid cooling and a connection to Entergy’s electrical grid, carrying Prometheus’s network, reliability, power, and fleet-management work into a larger purpose-built site.