NVIDIA's NeMo Switchyard Cuts Agent AI Costs by 74% With Smart Model Routing
NVIDIA's open-source NeMo Switchyard routes each agent workflow step to the right model automatically, cutting costs by up to 74% without rebuilding your app
- NVIDIA released NeMo Switchyard, a free open-source library that automatically routes each agent workflow step to the most cost-efficient model.
- 74% cost reduction measured by LangChain on 145 multi-turn tasks, sending only 7% of calls to a frontier model with ~6-point accuracy tradeoff.
- Three tuning-free routers (LLM classifier, stage router, escalation router) work out of the box; a tunable prefill router uses model internals for learned routing.
- Compatible with OpenAI, Anthropic, and Responses APIs, so existing agents can point at the Switchyard server with minimal code changes.
- Pre-alpha software: API and algorithms will change significantly before v1.0; not recommended for production yet without careful evaluation.
- Broad ecosystem: integrations with LangChain, LiteLLM, Kong, Cognition (Devin), Ramp, Cadence, and Siemens already live or in progress.
NVIDIA NeMo Switchyard is a new open-source library that solves one of the most expensive problems in production agentic AI: every step in an agent workflow gets sent to the same frontier model, even when a much cheaper one would do. Long-running agents spend most of their time on tool calls, result validation, and subagent delegation, and sending every one of those steps to a frontier reasoning model adds cost and latency. Switchyard fixes this by routing each step to the model best suited for it, automatically.
NeMo Switchyard is an open-source model routing library for AI agents that routes prompts to the most capable and efficient model for each step of an agent workflow automatically, based on specific needs. It ships alongside Nemotron 3.5 Lightning, a new 30B mixture-of-experts model with only 3B active parameters, purpose-built for the high-volume execution layer of agent pipelines.
The problem every agent builder hits
Some models are better for coding, some for reasoning, some for lightweight tasks, and some are optimized to run locally for greater privacy and efficiency. If you rely on one default model, you might either overspend or lose quality; if you manage routing manually, it becomes integration work that can slow down a deployment. Switchyard is the layer that makes this automatic.
The core insight is that no single model wins on every task. While DeepSeek V4 has the highest overall accuracy on the Terminal-Bench Hard benchmark, it is not the best model for every task group. Kimi K2.6 is better suited to ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science. A good router exploits these complementary strengths dynamically.
How it works under the hood
Switchyard is built around a provider-agnostic SDK called