Epoch's IPI Shows Nvidia GB300 Delivers 25x an H100's Real Inference Capacity
Epoch AI's new Inference Performance Index scores chips by memory and interconnect, not just FLOP/s, revealing GB300 serves 25x H100's load.
- Epoch AI launched the Inference Performance Index, scoring chips by real inference capacity, not FLOP/s.
- FLOP/s alone understates GB300's inference advantage over H100 by roughly 10x.
- Scores in GB300 equivalents: H200 at 0.12, B300 at 0.54, MI355X at 0.55, VR200 at 2.18.
- IPI integrates capacity curves across 50-400 tokens/s/req using Nvidia's AISimulate speed-of-light projections.
- Workload models long-context agentic inference with 262K cached tokens, averaged across four model architectures.
- Full methodology and code at GitHub; interactive rankings in Epoch's hardware explorer.
Peak floating-point operations per second can badly misstate an AI chip’s inference capacity. Epoch AI’s IPI estimates that an Nvidia GB300 delivers about 25 times an H100’s capacity per GPU on the modeled agentic workloads, despite having only 2.5 times its 8-bit arithmetic throughput. The capacity gain is about ten times larger than the raw-compute gain.
The bandwidth behind the 25× gap
During inference, chips spend time moving data as well as performing arithmetic. Long-context agents maintain a key-value cache containing attention data from earlier tokens, and the model reads that cache from high-bandwidth memory during each decoding step. When inference spans multiple GPUs, inter-chip links also carry model and cache data. FLOP/s captures the arithmetic ceiling while omitting these memory and communication limits.
One score from a family of curves
Epoch converts each chip’s capacity curve into a single Inference Performance Index score expressed in GB300 equivalents, or GB300e. GB300 NVL72 receives a score of 1 by definition. A score of 2 means the hardware can serve twice as many concurrent requests per GPU, averaged across the index’s workloads and output-speed requirements.
| System | Score, GB300e | Interpretation |
|---|---|---|
| GB300 NVL72 | 1.00 | Reference system |
| H200 HGX | 0.12 | GB300 provides roughly eight times the modeled capacity |
| B300 HGX | 0.54 | GB300 provides about 1.9 times the modeled capacity |
| MI355X | 0.55 | Similar modeled capacity to B300 HGX |
| VR200 | 2.18 | More than twice GB300’s modeled capacity |
| MI455X | 2.89 | Highest score in the Version 1 results |
Packaging moves the score
GB300 NVL72 reaches about 1.9 times B300 HGX’s modeled capacity per GPU even though both systems use Blackwell Ultra GPUs. NVL72 connects 72 GPUs within a high-bandwidth scale-up domain, while HGX connects groups of eight. A scale-up domain is the set of accelerators that exchange data over a system’s fastest local fabric, so the larger group lets more GPUs coordinate without sending traffic through slower network tiers.
More throughput can slow each response
Serving more requests in parallel amortizes each pass over model weights, increasing total throughput while reducing the output rate for each request. In Epoch’s example, raising concurrency from two to eight requests per chip lowers output speed from about 210 to 80 tokens per second per request. Total throughput rises from 420 to 640 tokens per second, an increase of roughly 50%.
For each chip, IPI integrates the area under the full capacity curve across output speeds from 50 to 400 tokens per second per request. The logarithmic speed axis gives equal weight to each doubling, preventing either the low-speed or high-speed end from dominating the score.
Inside the modeled ceiling
Epoch builds IPI with Nvidia’s AISimulate, which projects performance from arithmetic throughput in BF16, FP8, and FP4 formats; high-bandwidth memory capacity and bandwidth; and interconnect bandwidth and topology. The resulting speed-of-light estimate describes an optimal ceiling under the modeled hardware constraints. Production systems incur additional software and scheduling overhead.
Version 1 models a 262,144-token cached context and adds 1,024 input tokens during each prefill, the processing stage before token generation begins. It also includes speculative decoding, where a smaller or faster process proposes tokens for the target model to verify. These assumptions represent agents that retain long histories and repeatedly generate new output.
Scores average results across four architectures: Kimi K3, Qwen3-Coder-480B, GLM-5.2, and a proxy for MAI-Base-2. Their different attention mechanisms place different demands on memory capacity, bandwidth, and communication.
Epoch cites OpenRouter data from late September 2026 showing that agentic API keys consumed roughly seven times as many tokens as human API keys. That usage pattern explains the index’s emphasis on long contexts and sustained generation.
Where the estimate can drift
IPI estimates optimal hardware capacity under a specific workload. Kernel quality, compiler support, batching policies, parallelism strategies, networking, and software maturity determine how much capacity a production deployment realizes. Results can also shift with shorter contexts, different model architectures, output requirements outside the 50-to-400-token range, or different serving stacks.
Epoch compared predicted and measured Kimi K3 throughput on GB300 at five frontier operating points spanning 50 to 400 tokens per second per request. Predictions set to 50% of the modeled ceiling differed from measurements by an average absolute 12%, providing a rough calibration for the gap between the analytical ceiling and current kernels.
A calibration based on one model and chip family cannot establish the same error bound for every system in the ranking. Fleet decisions also require price, power consumption, availability, reliability, and deployment constraints, none of which the capacity score captures.
Epoch publishes the methodology and code and maintains a hardware explorer with the full rankings. IPI provides a useful first-pass estimate for agent-serving capacity when the target workload resembles its assumptions; final sizing still requires benchmarks using the intended model, context distribution, latency target, and serving stack.
Comments
