Model Routing Layers and the Prompt Cache Tax

The pitch for routing is a simple one, and it's why Cursor and Ramp both shipped a router this month. Send the easy turns to a cheap model, keep the hard ones on a frontier model, and the bill comes down without the output getting worse.

But Prompt caches are where that math starts to come apart. A cache belongs to one model, so the moment you switch, the model you switched to has to read the whole conversation again, and you pay full price for every token of it. So it’s not that simple, here is what we found in brief:

• Claude Sonnet 5 took over a 60,000-token coding session for me mid-way through, and that one turn billed $0.223. The next turn ran on the same model, in the same conversation, and it cost $0.020.

• Session size decides how much that hurts you. A switch costs about five times a warm turn when you're carrying 10,000 tokens, and nearly eight times when you're carrying 60,000.

• Layers are what I'd optimise after running these tests, rather than frequency. Which model handles a request matters less than which part of your system owns that choice, and the rest of this maps the three layers, the two locks that outrank cost, the tokenizer gap that breaks price comparison, and nine changes worth making on Monday.

Routing became infrastructure this month

Cursor Router went generally available for Teams and Enterprise on 22 July, and Ramp opened its router to outside users two days earlier. Model selection stopped being a preference in a dropdown.

Waste on that scale is real. Cursor says roughly 60% of its developers pick a single daily-driver model, which prices routine work at frontier rates.

Both companies put numbers on the fix, and both numbers are their own. Cursor reports 60% savings at frontier quality in its online tests and 30% to 50% for three early-access accounts, Ramp's chief technology officer cites a 30% cut, and nobody outside either company has reproduced any of it.

Claude Opus 5 then shipped on 24 July, two days after Cursor's launch, and as of 27 July Cursor's published model table still listed Opus 4.8 and Fable 5 with no Opus 5 entry.

That staleness was anticipated. Cursor says the classifier was built "for a world in which updated models get shipped early and often," and my read is that the instinct is right and also the whole difficulty, because a router is a bet on a portfolio that moves faster than the bet can be retrained.

The unit of routing is wrong

An agent session is not a sequence of independent prompts. It plans, calls a tool, receives output, gets told "continue," and fixes what broke.

Those short turns carry no meaning alone. "Run that again" is content-free without the forty turns above it, which is exactly why a classifier scoring the latest message reads it as cheap work.

Work from the vLLM Semantic Router team and AMD framed the second question a router has to answer: not only which model for this request, but whether switching inside this session is safe at all. Across 21,600 deterministic turns they cut model switches by 79.29% and eliminated 3,836 unsafe ones, with tool-loop violations falling from 3,404 to zero.

Their recommended policy still costs quality, at a delta of −0.0453 against −0.1433 for pinning one model forever, which is the kind of number vendors tend not to publish about themselves.

Cursor's router runs its classifier on each agent request in Balance and Intelligence modes, per its documentation. That is the per-request bet, stated plainly by the people making it.

What a switch actually costs

Anthropic publishes the multipliers: a cache read costs 0.1x the input price and a five-minute cache write costs 1.25x. Staying put is a 90% discount on everything your agent has already said.