Epoch AI Warns Token Demand Is Outpacing Global GPU Supply by 10x
Epoch AI models every Blackwell GPU on Earth and finds token demand growing 10x/year while supply grows just 3-4x — a crunch may already be here
- Supply vs. demand gap: Epoch AI finds token demand growing ~10x/year while global inference supply grows only 3-4x/year.
- Global capacity estimate: All Blackwell GPUs combined can serve 500M to 20B output tokens/sec depending on context length.
- Context length is the killer: Moving from 8k to 128k input tokens cuts throughput by ~40x, threatening agentic coding workloads most.
- Demand proxies are alarming: Meta employees used 60T tokens/month; Apple engineers reportedly spend $300/day on tokens each.
- Methodology: Model calibrated against 111 real inference runs from SemiAnalysis InferenceX, finding 65% compute and 30% bandwidth efficiency in practice.
- Everyday users may be shielded: Fast inference efficiency gains mean smaller, cheaper models will quickly match today's frontier capabilities.
The AI industry has spent the last two years in a frenzy of hardware investment, but a new analysis from Epoch AI asks the uncomfortable question: is all that silicon actually enough? Researchers Luke Emberson and Jaime Sevilla built a bottom-up model of global inference capacity and stacked it against the best available proxies for token demand. The verdict is sobering , demand appears to be outrunning supply by a wide margin, and the gap is widening.
The trillion-dollar question
Hyperscalers have been racing to build massive data centers, spending hundreds of billions in the process. The St. Louis Fed estimates that AI-related investment contributed about 1 percentage point , nearly 40% of the total , to US real GDP growth in the first three quarters of 2025, exceeding the IT investment contribution at the height of the dot-com boom. Whether the current AI buildout constitutes a bubble depends largely on whether there will be sufficient demand for the computing infrastructure being built.
Epoch AI estimates supply is growing 3 to 4 times per year, while demand for tokens is growing roughly 10 times per year , suggesting a compute crunch is nearing, if not already here.
How they counted every token the world could serve
The analysis is grounded in a specific, concrete setup: serving Kimi K2.6 , a 1-trillion-parameter mixture-of-experts model with 32 billion active parameters , on all of the world's Nvidia GB200 and GB300 chips. That means 1.9 million and 1.5 million individual GPUs respectively, representing roughly 40% of aggregate global compute on a FLOP/s basis. Three request profiles were modeled:
- General usage: 8,000 input tokens, 1,000 output tokens
- Agentic (medium): 25,000 input tokens, 1,000 output tokens
- Agentic (long): 128,000 input tokens, 1,000 output tokens
The researchers built a theoretical model of inference from first principles, then calibrated it against real-world data from SemiAnalysis's InferenceX dashboard , a repository of thousands of real inference experiments. Calibration revealed 65% compute efficiency and 30% bandwidth efficiency in practice, significantly below peak specs.

Prefill, decode, and the two tricks that matter
The model breaks inference into its two core phases. During prefill, all input tokens are processed in parallel to populate the KV cache (a memory structure that stores intermediate attention computations so they don't need to be recomputed). During decode, output tokens are generated one at a time. These two phases have very different bottlenecks: prefill is compute-bound (the GPU math units are the limiting factor), while decode is bandwidth-bound (the bottleneck is how fast weights and the KV cache can be loaded from memory).

Two optimization techniques are layered on top of this baseline:
- Chunked prefill: Because prefill is compute-bound and decode is bandwidth-bound, the two phases leave complementary resources idle. Chunked prefill interleaves them , while one batch is decoding (using bandwidth), the next batch's prefill computations run on the spare compute units. The result is that total cycle time collapses to the maximum of total compute work or total bandwidth work, not the sum.
- Speculative decoding: A small "draft" model proposes several tokens ahead, and the main model verifies them all in a single forward pass. When the draft is right, you get multiple output tokens per pass. Using state-of-the-art implementations, this boosts decoding throughput at a fixed batch size by roughly 3x, with a combined effect on total throughput of 1.6 to 2x.
The numbers: how much can the world actually serve?
The final estimate suggests the world's Blackwell GPUs could currently deliver a combined 500 million to 20 billion output tokens per second, or between 150,000 and 7 million tokens per month for each person on Earth. The wide range is driven almost entirely by context length , longer inputs crush throughput because attention costs scale quadratically with sequence length.
| Request type (input:output) | Output tokens per second (global) |
|---|---|
| 8k:1k (general) | ~20 billion |
| 25k:1k (agentic, medium) | ~6 billion |
| 128k:1k (agentic, long) | ~500 million |
Google, likely the most avid token producer today through Search summaries, recently claimed it was serving 1.2 billion tokens per second across all its platforms. Assuming an 8