NVIDIA's NeMo Switchyard Cuts Agent AI Costs by 74% With Smart Model Routing

NVIDIA NeMo Switchyard is a new open-source library that solves one of the most expensive problems in production agentic AI: every step in an agent workflow gets sent to the same frontier model, even when a much cheaper one would do. Long-running agents spend most of their time on tool calls, result validation, and subagent delegation, and sending every one of those steps to a frontier reasoning model adds cost and latency. Switchyard fixes this by routing each step to the model best suited for it, automatically.

NeMo Switchyard is an open-source model routing library for AI agents that routes prompts to the most capable and efficient model for each step of an agent workflow automatically, based on specific needs. It ships alongside Nemotron 3.5 Lightning, a new 30B mixture-of-experts model with only 3B active parameters, purpose-built for the high-volume execution layer of agent pipelines.

The problem every agent builder hits

Some models are better for coding, some for reasoning, some for lightweight tasks, and some are optimized to run locally for greater privacy and efficiency. If you rely on one default model, you might either overspend or lose quality; if you manage routing manually, it becomes integration work that can slow down a deployment. Switchyard is the layer that makes this automatic.

The core insight is that no single model wins on every task. While DeepSeek V4 has the highest overall accuracy on the Terminal-Bench Hard benchmark, it is not the best model for every task group. Kimi K2.6 is better suited to ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science. A good router exploits these complementary strengths dynamically.

How it works under the hood

Switchyard is built around a provider-agnostic SDK called switchyard-libsy. Each model target has a semantic name, while the client behind it maps that name to the provider endpoint and model ID. This separation keeps the routing logic independent of a specific provider. Swap out a model or move it to a different endpoint, and your routing logic stays intact.

The library offers two families of routers: tuning-free and tunable. The tuning-free ones work out of the box with no training data required:

• LLM classifier: Uses an LLM-as-judge to pick a model for a request, then maintains session affinity with that model across subsequent turns so it does not re-classify work that has not meaningfully changed.
• Stage router: Designed for coding agents. It examines recent tool activity to decide how much model capability the agent needs. Severe errors, repeated unproductive work, or prolonged exploration push the turn toward the capable model. Steady writes and edits, especially once tests pass, favor the efficient model.
• Escalation router: Starts each conversation with a lower-cost model. An LLM judge monitors the progress of the task, turn by turn, and moves the session to a more capable model when it detects sustained difficulty.

The tunable prefill router goes deeper. During training, it extracts the LLM's residual stream (the internal activations of the model) to estimate query complexity. A shared-trunk MLP maps those signals to accuracy labels for each LLM in the routing pool. At inference time, the prefill states act as input to the router, and the shared trunk predicts the likelihood that each LLM will successfully complete the task. This is the most powerful option, but it requires labeled workload data to train.

The numbers that matter

The benchmark results from real partners are striking. Financial platform Ramp reduced task runtime by 33% and costs by 58%, while LangChain slashed multi-turn agent costs by 74% with negligible impact on accuracy.

LangChain's test is worth unpacking. They ran their Deep Agents evaluation suite through Switchyard and measured how many turns the router sent to a frontier model. The answer was 7%. A 30B parameter model handled the other 93%. Routing between Nemotron 3.5 Lightning and Claude Opus 4.8 cut the total cost by 74% against running Opus alone, while retaining 93% of its accuracy for the same calls.

Cognition tested staged routing inside Devin Desktop on their FrontierCode benchmark. The result was near-frontier performance at 50.6% accuracy at a $3.11 mean cost, within 2.8 percentage points of Opus 5 accuracy at approximately 28% lower mean cost.

What it is good at, and where it falls short

Switchyard shines in multi-turn, long-running agent workloads where the mix of easy and hard steps is predictable. It is particularly well-suited for:

• Coding agents that alternate between exploration and mechanical implementation
• Customer support pipelines where most turns are routine but occasional complex cases need escalation
• Multi-step retrieval and tool-use workflows
• Enterprise automation where cost predictability matters

But it is not a drop-in solution. Model routing is not plug-and-play. Like a network router, getting the best outcomes requires skilled operators and careful configuration. Existing agents will likely need re-engineering to benefit from routing. You also have to choose the right agent harnesses and deployment topology. Switchyard's ROI depends entirely on these architectural decisions upstream. Get those wrong, and the router becomes overhead rather than optimization.

There is also a maturity caveat. The GitHub repo labels Switchyard as pre-alpha software that is evolving rapidly, with the API and algorithms expected to change significantly before reaching v1.0. The learning-based prefill router also appears to be only conceptually introduced at the time of writing and not yet included in the release.

The bigger picture

Model routing itself is not new. OpenRouter, AWS Bedrock Agent Core, and Microsoft Foundry already offer cloud-based versions. What matters is that NVIDIA is making this open source and available to run locally. That architectural choice changes the game for hybrid AI deployments.

Switchyard sends each prompt in an agent workflow to the most capable and efficient model for that step, across a developer's own mix of open, proprietary, and NVIDIA models, without requiring the application to be rewritten. Developers can tune or swap the routing algorithm to match their priorities on quality, latency, or cost.

The ecosystem integration is already broad. Partners include LangChain, LiteLLM, Kong, Cognition, Ramp, Nous Research, Cadence, and Siemens, with Switchyard plugging into existing agent frameworks and LLM gateways. The reference server accepts OpenAI, Anthropic, and Responses API requests out of the box, so pointing an existing agent at it requires minimal changes.

To get started, install the CLI and point your agent's base_url at the Switchyard server:

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"

The full code and routing algorithm docs are available on the NVIDIA-NeMo/Switchyard GitHub repo. Given the pre-alpha status, treat it as a powerful experiment for now, but one with real production results already behind it.