NVIDIA's TensorRT Model Connect Ships Hugging Face Models to C++ in Two Commands
NVIDIA's new open-source tool skips ONNX entirely, turning Hugging Face checkpoints into PyTorch-free C++ TensorRT bundles with two CLI commands.
- NVIDIA released TensorRT Model Connect in public preview under Apache-2.0.
- Two commands (
trtmc build,trtmc run) go from Hugging Face checkpoint to TensorRT inference. - No intermediate ONNX export; produces a versioned
.bundleloadable from native C++. - C++ task APIs include generate, transcribe, generate_image, embed, and solve, PyTorch-free at runtime.
- GB300 snapshot: 105 profiles across 76 families, 102 beating reference by more than 5%.
- Entire project built by OpenAI Codex agents under human review; Linux aarch64 wheels only for now.
NVIDIA just dropped a tool that collapses one of the more painful workflows in production ML: getting a Hugging Face model into a native C++ inference binary. TensorRT Model Connect (TRTMC) is now in public preview, and it promises to take you from a Hugging Face checkpoint to end-to-end TensorRT inference in two commands, with no ONNX detour in between.
It's an open-source project that takes a supported Hugging Face or local checkpoint to end-to-end TensorRT inference in two commands, and the project is Apache-2.0 licensed and ships as a collection of family-owned reference implementations rather than a single generic converter. There's also an unusual footnote on how it was built, which we'll get to.
The two-command pitch
The whole workflow is a build step followed by a run step. Here's the quick start for Qwen3-0.6B straight from the docs:
trtmc build Qwen/Qwen3-0.6B --precision bf16 --max-cache-length 16384 --output qwen3-0.6b.bundle
trtmc run ./qwen3-0.6b.bundle --prompt "What is the capital of France? Answer in one word." --chat-template --no-thinkingThe output is a versioned .bundle file, and that same artifact loads directly from C++ via trtmc::load("./qwen3-0.6b.bundle"). The build produces a versioned .bundle artifact that runs through native C++ task APIs, so inference can execute in a C++ service, embedded application, or robotics stack without PyTorch in the runtime path.