NVIDIA's Groq 3 LPX Hits 4x Faster Inference at 100K Context

Artificial Analysis just published the first independent benchmark of NVIDIA's Groq 3 LPX rack, and the numbers reset expectations for what long-context inference can feel like. On a private endpoint serving Gemma 4 31B, the system generated 3,431 output tokens per second at 100K input context, with essentially no drop-off from the 10K result of 3,382 tokens per second.

NVIDIA stood up the system in its own data centers and gave Artificial Analysis early access ahead of full-scale production, which the company says will enter operation later this year. For comparison, the fastest public endpoint on the same Gemma 4 31B 100K benchmark clocked 870 tokens per second, meaning the LPX rack delivers roughly 4x the throughput of anything you can rent today.

Why interactivity at 100K context is a hard problem

Agentic workflows don't stay short. At the end of each turn, the agent's output is appended to the continually growing context that is fed into all subsequent turns, and by turn 200 the running context routinely climbs into the hundreds of thousands of tokens.

Pushing more than 3,000 tokens per second per user while holding a 100K KV cache forces you into very small batch sizes. That's where tensor parallelism, the standard divide-and-conquer trick for inference, starts to break down. Many small tensors must transfer between compute units, with each tensor leaving and arriving exactly where and when it needs to. Time spent communicating and combining results can easily become comparable to, or larger than, the time saved by distributing the computation.

The dominant cost in this regime is first-bit latency, the fixed startup time paid on every transfer to figure out which link to use, sync endpoints, and arbitrate conflicts. Bandwidth is a secondary concern.

The compiler sees everything

Groq 3 LPX uses a deterministic execution model, which is where the design differs meaningfully from a conventional GPU rack. The compiler has full visibility into every compute unit across the 256 LP30 local processing units (LPUs), the 128 GB of total SRAM-based memory collectively in those chips, and the 96 C2C links per chip running at 112 Gbps each. From that view it produces a cycle-accurate schedule of the entire workload before execution begins.

Two consequences flow from this:

• No real-time arbitration. Because the compiler already knows when every packet will move on every link, LPUs can send data on the exact clock cycle it's ready and consume it on the exact cycle it arrives. Links are point-to-point between LPU pairs, and any LPU can act as a router.
• Fine-grained overlap of compute and communication. The compiler schedules workloads on its compute and communication units at the level of 320-byte vectors. Since a matmul is just a stack of dot products, an LPU can begin transmitting the first 320-column slice of the output the moment it's computed, instead of waiting for the whole matrix operation to finish.

The net effect is that tensor parallelism actually pays off at batch size 1, which is the operating point that agentic workloads live in.

The benchmark numbers

Artificial Analysis ran its standard suite at 1K, 10K, and 100K input sequence lengths, measured over 50 sequential single-concurrency requests. Output speed barely moved between context lengths, which is the more interesting finding. On most systems, 100K context tanks throughput. Here the delta between 10K and 100K was under 50 tokens per second.

NVIDIA also ran the open-source SPEED-Bench coding benchmark on the same setup and reported a median of 4,767 output tokens per second and a P80 of 5,520 output tokens per second on Gemma 4. Joint testing by Artificial Analysis and NVIDIA confirmed that the NVIDIA configuration for both benchmarks produced no loss of precision or model quality in its outputs.

In developer terms: decoding 5,000 tokens takes about 1.5 seconds versus 50 seconds at 100 tokens per second. Today's popular agentic coding tools run closer to 60 tokens per second, so the practical gap is even larger.

Where LPX slots into Vera Rubin

Groq 3 LPX isn't a standalone product you'd deploy on its own. It works as the low-latency serving tier alongside Vera Rubin NVL72, which handles the heavy lifting for larger models. The pairing supports three co-execution patterns:

1. Prefill-decode disaggregation. Vera Rubin NVL72 does the prefill, hands the KV cache to LPX once per turn, and LPX runs the decode from SRAM.
2. Attention-FFN disaggregation. Vera Rubin computes attention and holds the KV cache in DRAM while LPX executes the FFN layers, with only intermediate tokens crossing the link.
3. External-drafter speculative decoding. LPX runs a small draft model ahead of the large target model on Vera Rubin, which verifies and commits tokens. Only draft tokens cross the link.

NVIDIA projects that this configuration scales to models in the 2T-parameter range, which is the target for the next generation of multi-agent systems.

What this changes

Two assumptions worth updating. First, the ceiling on interactive inference at long context is set by how tightly you can schedule communication between chips, not by model size. Ahead-of-time compiler scheduling can beat runtime arbitration by a factor of several at small batch sizes. Second, sub-second responses at 100K context are now a design point rather than an aspiration, which changes what's realistic for agentic coding tools, long-document analysis, and multi-turn sessions that would otherwise force you to summarize and lose fidelity. For anyone benchmarking against Artificial Analysis leaderboards, the public-endpoint ceiling is about to move.