ZML Runs Inference 2x Faster Than TensorRT-LLM Without a Single Line of Python
A Zig-based inference stack compiles LLMs into standalone binaries that run on NVIDIA, AMD, TPU, and Trainium without rewrites.
Takeaways
- ZML is a Zig-based inference stack compiling models via MLIR and OpenXLA into native binaries.
- One codebase targets NVIDIA CUDA, AMD ROCm, Google TPU, AWS Trainium and CPU via Bazel flags.
- Supports Llama 3.1/3.2, Qwen 3.5, and LFM 2.5, loading from Hugging Face, S3, or local paths.
- LLMD server ships OpenAI-compatible API, continuous batching, paged attention in a 2.4 GB image.
- Team claims 2x faster than TensorRT-LLM on H100; still alpha with build-your-own-stack positioning.
- Apache 2.0 licensed, 3.6k GitHub stars, 93.5% Zig code, backed by Paris startup with 20 engineers.
Most inference stacks today are Python wrappers sitting on top of CUDA kernels, which works fine until you try to run the same model on AMD or a TPU. ZML takes a different route. Written in Zig, it compiles model graphs into native binaries that target any accelerator from a single source.
The project has crossed 3,600 stars on GitHub and sits on a stack of Zig, MLIR, OpenXLA, and Bazel. Where vLLM, Ollama, and most other inference servers wrap Python around CUDA kernels, ZML compiles model computation graphs into standalone native binaries. The runtime has zero Python dependencies, minimal memory overhead, and direct hardware access.
One binary, every accelerator
Hardware portability without rewrites is the pitch. Switching backends is a Bazel flag, not a codebase migration:
--@zml//platforms:cuda=truefor NVIDIA--@zml//platforms:rocm=truefor AMD--@zml//platforms:tpu=truefor Google TPU--@zml//platforms:neuron=truefor AWS Trainium and Inferentia 2
A handful of build flags produce a Docker image with a Linux binary that runs across NVIDIA CUDA, AMD ROCm, Google TPU, AWS Neuron, or CPU runtimes, and a single command pushes the image to a container registry. For teams tired of maintaining separate deployment paths per vendor, that is the headline feature.