Inception Labs' Mercury 2 Hits 1,000 Tokens per Second on Azure

Mercury 2, the diffusion-based language model from Inception Labs, is now live on Azure AI Foundry. The pitch is simple but striking: reasoning-quality output at over 1,000 tokens per second, at less than half the cost of comparable models. That's not a minor speed bump. It's a different architecture entirely.

Why the typewriter metaphor finally breaks

Every major language model you've used, GPT, Claude, Gemini, Llama, generates text the same way: one token at a time, left to right, each token depending on all the ones before it. It works, but it creates a hard ceiling on speed. Mercury 2 generates all tokens simultaneously and refines them through iterative denoising passes, a process borrowed from image generation models like Stable Diffusion.

The core insight is that text diffusion can work with discrete tokens using masking as the "noise" process, sometimes called discrete diffusion or absorbing diffusion, which sidesteps issues with continuous diffusion applied to language. In practice: rather than adding Gaussian noise to continuous embeddings, Mercury 2 uses a masking-based corruption process designed specifically for discrete tokens, producing more stable training and sharper convergence during inference.

Think of it less like a typewriter and more like an editor. The model starts with a canvas of masked tokens, then refines the whole draft in parallel, converging over a small number of steps. The number of denoising steps adjusts dynamically based on output complexity.

The numbers that matter

Mercury 2's standout claim is raw throughput: roughly 1,000 tokens per second versus about 89 tokens/sec for Anthropic's Claude Haiku 4.5 Reasoning and 71 tokens/sec for OpenAI's GPT-5 Mini. That's not 5x faster than slow frontier models. It's 5x faster than models already optimized for speed.

On quality, Mercury 2 scored 91.1 on AIME 2025, 73.6 on GPQA, 71.3 on IFBench, 67.3 on LiveCodeBench, 38.4 on SciCode, and 52.9 on Tau2. These scores place Mercury 2 within competitive range of Claude 4.5 Haiku and GPT 5.2 Mini on quality, while delivering roughly 10x the throughput.

The full spec for the Azure deployment:

• Speed: 1,009 tokens/sec on NVIDIA Blackwell GPUs
• Price: $0.25/1M input tokens, $0.75/1M output tokens
• Context: 128K token window
• Features: tunable reasoning, native tool use, schema-aligned JSON output, OpenAI-compatible API
• Compliance: SOC2, HIPAA via Azure infrastructure

Real production numbers, not just benchmarks

The speed claims hold outside the lab. Augment Code uses Mercury 2 for context compaction, and compared to Claude Opus 4.7, switching led to an 82% drop in latency and a 90% reduction in cost, while maintaining the exact same compaction quality. The compaction step dropped from roughly 150 seconds to 27 seconds and became fast enough to be invisible.

Augment also uses Mercury 2 as the high-speed planner for their Prism router, helping direct tasks to the most efficient model. Prism's dynamic routing reduces total LLM spend by 30% without sacrificing code quality. Other early adopters include Wispr Flow, OpenCall, and Happyverse AI for voice applications, and the Zed editor for coding workflows.

Where it shines, and where it doesn't

Mercury 2 is purpose-built for latency-sensitive workloads. The four use cases that benefit most:

• Agentic loops: Multi-step agent workflows chain dozens of inference calls per task. Cutting latency per call doesn't just save time, it changes how many reasoning steps you can afford to run.
• Real-time voice: Voice interfaces have the tightest latency budgets in AI. Mercury 2 makes reasoning-quality output viable within natural speech cadences.
• Coding and editing: Autocomplete, next-edit suggestions, and interactive code agents where any pause breaks developer flow.
• Search and RAG pipelines: Multi-hop retrieval and reranking latencies stack fast. Mercury 2 lets you add reasoning to the search loop without blowing your budget.

The tradeoffs are real though. Quality trails by 5-15% on complex reasoning benchmarks compared to frontier autoregressive models, though it matches on structured output and translation tasks. Autoregressive models can start streaming output almost immediately, while diffusion models may need to run several denoising passes before producing any visible output, so it's worth checking current API behavior if time-to-first-token matters for your use case. The tooling, fine-tuning support, and community resources around Mercury 2 are also thinner than what you'd find for GPT-4o or Claude.

The broader shift this signals

Inception was founded by Stanford, UCLA, and Cornell researchers behind foundational diffusion work. CEO Stefano Ermon authored the text diffusion paper that won Best Paper at ICML 2024. The startup raised $50 million with backing from NVIDIA's venture arm and investors Andrew Ng and Andrej Karpathy.

The dominance of autoregressive models isn't because the architecture is theoretically optimal. It's largely because the training recipe is well-understood, the scaling laws are predictable, and enormous amounts of infrastructure have been built around it. Mercury 2's Azure landing changes the calculus for production teams: you no longer need to host it yourself or accept a niche API. It's now available inside the same enterprise environment where many teams already run their workloads.

Scaling laws for diffusion LLMs aren't yet established. The autoregressive scaling curve has been mapped over years. Inception Labs and other research groups are still determining whether diffusion models follow similar scaling patterns or require different optimization strategies. That's the open question. But for the specific class of high-throughput, latency-sensitive production workloads, the architecture already delivers.

Getting started on Azure

The model is available now in US and Canada regions through the Azure AI Foundry model catalog. The API is OpenAI-compatible, so dropping it into an existing codebase is straightforward. Inception recommends ND-H100-v5 instances for optimal speeds. A minimal Python call looks like this:

import requests, json
url = "https://<your-endpoint>.inference.ml.azure.com/v1/chat/completions"
headers = {
    'azureml-model-deployment': 'mercury-2',
    'Content-Type': 'application/json',
    'Authorization': 'Bearer <your-api-key>'
}
payload = json.dumps({
    "model": "mercury-2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": True,
    "temperature": 0.0,
    "max_tokens": 512
})
response = requests.post(url, headers=headers, data=payload)
print(response.json())

The Mercury 2 software license is billed at a flat hourly rate, with compute costs billed separately through your Azure account. For teams already invested in the Azure ecosystem, this is the lowest-friction path to testing whether diffusion-based generation actually changes the feel of your product. The official documentation covers integration guides and benchmark details.