CMU Finds JEV Matches GPT-6 Accuracy at 277 Times Lower Cost
A CMU study shows a cheap decision-only judge matches frontier LLMs on easy calls while routing hard ones upstream, retaining 99% accuracy at 57% cost.
- CMU studies TypeSafe's JEV decision-only judge against 16 generative and reward-model judges
- JEV lands within 3 points of GPT-6 on preference and factuality at 0.36% the fee
- Falls 14.6 points behind on JudgeBench derivations and 19.8 on style-adversarial pairs
- Confidence cascade retains 99% of GPT-6 accuracy at 57% of cost
- Median latency 0.152s vs 1.885s for GPT-6; $0.044 vs $12.18 per 1,000 judgments
- Reference-free prose evaluation defeats every tested judge, not JEV alone
A low-cost judge can route routine LLM evaluations
Teams increasingly use language models to select responses, audit training data, and run benchmarks. Each judgment adds tokens, latency, and cost to the development loop, especially when a frontier reasoning model grades every item. Carnegie Mellon University researchers tested whether a decision-only service could handle routine cases and escalate uncertain ones to a stronger model.
The JEV-as-a-Judge paper evaluates TypeSafe AI’s hosted judging service, which returns a typed verdict and label probabilities without a written rationale. The authors compare it with 16 generative and reward-model judges across preference, factuality, and answer-adjudication tasks, then use blinded human review to resolve disagreements. On preference and evidence-grounded factuality, JEV finishes within three percentage points of GPT-6 while costing 0.36% as much on the tested workload.
LLM grading has its own compute bill
Generative judges can interpret rubrics and work through comparisons, but sequential generation increases latency and token-based fees. Repeating that process for every response and model revision can make evaluation a substantial infrastructure expense.
Reported confidence creates a second engineering problem because language models often assign high confidence to incorrect verdicts. JEV exposes probabilities over its allowed labels, giving evaluation pipelines a numeric signal that can be validated and used for routing.
A typed verdict with usable confidence
TypeSafe JEV accepts structured state, natural-language instructions, and an allowed output type. The service supports three response formats:
- Choice: probabilities over a supplied set of labels
- Noul: a probability for a yes response
- Score: probabilities over ordered rubric levels
The experiments submit one Choice question per request and treat the highest label probability as the confidence score. Generative baselines receive the same requirement to return a decision and probabilities without a written explanation. Those baselines include GPT-4.1 mini, GPT-5.x, GPT-6 Astra, Claude Sonnet 5, Gemini 3, Qwen3, and GPT-OSS 120B.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
