Google Cloud Backs SGLang to Kill the GPU-to-TPU Migration Tax

Google Cloud and RadixArk have announced a partnership to bring SGLang, the open-source inference framework running on hundreds of thousands of GPUs worldwide, to Google's TPU hardware. The first deliverable, SGL-JAX, is available today. A second backend, SGL-torchtpu, arrives later this year and lets teams run the same SGLang serving stack on TPUs using standard PyTorch tooling, no JAX knowledge required.

The framework behind a trillion tokens a day

SGLang runs inference for Google, Microsoft, NVIDIA, Oracle, AMD, LinkedIn, and xAI, generating trillions of tokens per day across more than 400,000 GPUs as of 2026. Released in 2024 and developed by Ying Sheng and Banghua Zhu out of LMSYS, the Berkeley research collective behind Chatbot Arena, it has grown to over 30,000 GitHub stars and 1,700 contributors.

SGLang's commercial backer, RadixArk, launched with a $100 million seed round at a $400 million valuation. The round was led by Accel and co-led by Spark Capital, with participation from NVentures (NVIDIA's venture arm), AMD, John Schulman (OpenAI co-founder), Soumith Chintala (PyTorch creator), and Thomas Wolf (Hugging Face co-founder). The framework itself remains Apache 2.0 and free to self-host.

What's shipping and when

The partnership delivers two TPU backends targeting different developer profiles:

• SGL-JAX (available now): Uses JAX to compile the model's forward pass on top of SGLang's server architecture, delivering native TPU inference with continuous batching, prefix caching, tensor and expert parallelism, speculative decoding, and optimized TPU kernels. Supported models include Gemma, Qwen, DeepSeek, GLM, Kimi, Grok, and diffusion models like Wan and Flux.
• SGL-torchtpu (later this year): A PyTorch-native path built on Google's TorchTPU stack. TorchTPU supports divergent executions (MPMD, where different processes run different code simultaneously, unlike the single-program model TPUs traditionally required), isolating communication primitives for correctness while preserving XLA's ability to overlap communication and computation across a distributed TPU deployment. In practice: existing PyTorch code runs on TPUs without adopting JAX's functional programming model.

The team is also committing to Day 0 TPU support for new open models. Any model that ships with SGLang GPU support will simultaneously work on TPUs, extending to every new TPU generation going forward.

The migration tax problem

Google VP Bill Jia's phrase "migration tax" names the real friction. Moving a production inference workload from GPU to TPU today means rewriting serving code, swapping frameworks, revalidating outputs, and retraining your team. That overhead is why most teams stay on NVIDIA even when TPU economics look better on paper.

The economics of TPUs have become harder to ignore. Midjourney cut inference costs 65% after migrating from GPUs. Google's TPU v6e delivers 4.7x better price-performance for inference workloads and consumes 67% less power than equivalent GPU clusters. Those gains matter: inference consumes 80–90% of total compute spend over a model's lifetime, and for every $1 billion spent training a model, organizations typically face $15–20 billion in inference costs during production. The hardware choice at serving time carries real weight.

What SGLang brings to TPUs

Most TPU inference solutions today offer basic serving. SGLang brings a production-grade stack with features that compound at scale:

• Radix Cache: Stores KV data (intermediate computation saved during inference) in a tree structure where shared prefixes are kept once and reused by all requests with matching token sequences, cutting redundant computation for common prompts, system messages, and multi-turn conversations.
• HiCache: A three-tier caching hierarchy that treats GPU memory as L1, host memory as L2, and distributed storage as L3, dramatically expanding how much KV state stays warm across requests.
• Speculative decoding: A smaller draft model predicts multiple tokens ahead; the main model verifies them in parallel. The latest SGLang release shipped zero-overhead Spec V2 with an 11% end-to-end throughput improvement.
• Full parallelism support: Data, tensor, expert, context, and pipeline parallelism across multi-host TPU pods.
• Quantization: INT4 and FP8 for reduced memory and compute costs.

Who wins and who feels the pressure

TPU v8, announced recently, splits into two chips: TPU 8t for training and TPU 8i for inference, with 80% better inference cost versus the previous Ironwood generation. Google is pushing TPUs as a serious inference platform, and locking in the dominant open-source inference framework as a first-class citizen on that hardware is a direct play to pull workloads away from NVIDIA-centric clouds.

Teams already running SGLang on GPUs gain the most immediately: a credible, low-friction path to TPU economics without a framework rewrite. The open-source community also benefits. SGLang already carries day-0 support for virtually every major open model family, including Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, and Mistral, across NVIDIA GPUs, AMD GPUs, Intel CPUs, and Google TPUs. This partnership deepens that hardware-agnostic positioning.

The pressure lands on vLLM, the other major open-source inference framework. vLLM has its own TPU backend and has posted 2–5x performance improvements on it, numbers SGLang will now compete against directly with Google's backing. NVIDIA also feels the squeeze: every workload that migrates to TPUs is a GPU-hour lost.

What to watch next

The JAX backend is live at sgl-project/sglang-jax. The SGL-torchtpu backend is the one most teams will want. Once it ships, the choice between GPU and TPU becomes a cost and availability decision rather than a software one. The SGLang API stays identical across hardware. The only thing that changes is the bill.