OpenAI's Jalapeño Chip Beats Nvidia's Best on Inference by 3.6x

OpenAI has published the first performance numbers for Jalapeño, its custom inference chip built with Broadcom. The results are the clearest signal yet that OpenAI is serious about owning its silicon stack, and the numbers put the chip ahead of Nvidia's newest inference-optimized parts on several axes.

Jalapeño was first unveiled as OpenAI's inaugural custom inference processor, designed and manufactured in collaboration with Broadcom and built specifically for the shape of OpenAI's serving workloads. What is new this week is the first batch of measured performance data, plus a deployment timeline.

The headline numbers

OpenAI benchmarked Jalapeño on InferenceX using GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Every run was normalized by each accelerator's published power rating, since existing hardware often forces trade-offs between throughput and latency and OpenAI wanted apples-to-apples comparisons at matched user experience.

Across the three public models, Jalapeño delivered:

• 1.5 to 1.9 times more AI work per watt at peak throughput
• 1.7 to 3.6 times lower end-to-end latency
• 2.1 to 4.1 times higher performance on highly interactive workloads

The comparison points were Nvidia's GB200 (1,200 W) for GPT-OSS and GB300 (1,400 W) for the larger models. Jalapeño is rated at 700 watts, though its measured sustained power stayed at or below 550 watts on the workloads tested. On Kimi K2.5 1T, it hit roughly 1.5x higher peak performance per watt and 3.4x lower end-to-end latency than the reference GB300 system.

One chip, two very different jobs

The interesting engineering story is that Jalapeño refuses the classic tradeoff between throughput mode and interactive mode. Language model inference has two very different phases: prefill, which chews through the prompt and is compute-bound, and decode, which generates tokens one at a time and is memory-bandwidth-bound. Most systems are tuned for one and pay a penalty on the other.

OpenAI's approach was to co-design the chip, memory, network, and rack around the actual shape of these workloads. Model state, including the KV cache used during generation, can be explicitly pinned local to the compute, avoiding the long trips across the network that normally leave cores idle. The fabric is large enough that an entire workload can remain within one connected system, which keeps the whole request path fast.

The result is a single architecture that can serve batched, throughput-heavy queues and interactive, agent-style workloads. That matters for agents specifically, where many sequential steps mean latency compounds across a task.

Built by AI, for AI to program

Two details stand out about the development process. The engineering timeline moved from early schematics to fabrication readiness in nine months, when new processor cycles are typically measured in years. And OpenAI leaned heavily on its own models to get there.

Earlier generations of GPT helped with the physical design, including optimizing arithmetic circuits to squeeze more compute into the die. Newer models are writing kernels. According to VentureBeat's reporting, the companies attributed this speed to a deep software-hardware co-development process that actively used OpenAI's own models to accelerate parts of the chip design.

The programming model is deliberately explicit: engineers describe work through local tensors, explicit communication, and predictable synchronization. That structure makes it easier for both humans and code-writing models to schedule work across the chip. Using Codex with an internal model called GPT-Astra, the team brought three open-weight models that were not in the original production plan to high performance in two months. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than the ones written by human experts.

What it does not change

Jalapeño is inference-only. Training stays where it is. OpenAI was explicit that it will continue to deploy accelerators from Nvidia and other partners for both training and inference workloads. The chip is aimed at the operating cost side of the business, not the R&D side.

That is the interesting economic angle. Earlier reporting suggested roughly 50% lower cost per inference token versus Nvidia GPU clusters, which is the layer that determines API pricing, rate limits, and whether agent loops are economically viable when they fire dozens of tool calls per user session.

Deployment and roadmap

OpenAI plans to begin deploying Jalapeño inside its own compute infrastructure by year end. The chip is described as the first generation of a multigenerational platform, with the 10-gigawatt Broadcom buildout scheduled to complete by the end of 2029. That implies a cadence of successive chip generations over the next three years. Gen 2 is already deep in development, and Gen 3 is taking shape.

For anyone building on the OpenAI API, the practical implications are straightforward:

• Faster ChatGPT and Codex responses as Jalapeño ramps in production
• Cheaper per-token economics that could show up in API pricing over time
• Better headroom for agent workloads, where dozens of sequential inference calls stack up
• Reduced risk of capacity throttling as OpenAI adds a second supply lane alongside Nvidia

Custom silicon eats the stack

Jalapeño puts OpenAI in the same category as Google with TPUs and Amazon with Trainium and Inferentia: a model provider that also owns its silicon. The competitive question is whether custom ASICs, tightly co-designed with a specific family of models, can consistently beat Nvidia's general-purpose GPUs on inference cost per token. The results OpenAI just published say yes, at least for the workloads that matter most to its own business.

The assumption worth updating is that Nvidia's inference lead is durable simply because the software ecosystem is sticky. When the same company controls the model, the compiler, the kernels, and the chip, that stickiness works in reverse. Jalapeño is a bet that owning the full stack compounds into an efficiency advantage a general-purpose vendor cannot match.