Penn Researchers Show Jev Cuts AI Grading Costs by 325x
A new study pits TypeSafe's typed classifier against flash-tier LLM judges on rubric grading, finding huge speed and cost wins but correlated errors that break cascades.
- UPenn researchers Rao and Callison-Burch benchmarked TypeSafe's Jev classifier against three flash-tier LLM judges.
- Jev accuracy differed significantly in only 8 of 27 paired comparisons across nine rubric panels.
- LLM judges cost 29 to 325 times more and ran 30 to 220 times slower than Jev.
- On graded criteria, all four judges cluster together and score lower than humans.
- Cascading Jev's low-confidence calls to an LLM gains at most 1.5 accuracy points.
- Correlated errors mean the LLM repeats Jev's most confident mistakes, breaking the cascade idea.
An arXiv paper by Delip Rao and Chris Callison-Burch of the University of Pennsylvania tests whether rubric evaluations require a generative language model. Across nine benchmark panels, three flash-tier LLM judges cost 29 to 325 times as much as Jev and took 30 to 220 times as long. Accuracy differed significantly in eight of 27 pairwise comparisons, while a simulated confidence cascade improved on the best single judge by no more than 1.5 percentage points with cross-fitted thresholds.
Rubrics multiply the bill
Rubric-based evaluation turns an open-ended response into decisions about individual criteria. A chemistry grader might check whether an answer states a required relation, while an essay grader might assign one of five grammar levels. Benchmarks such as HealthBench use these judgments, and reinforcement-learning systems can use them as reward signals.
Per-criterion graders make one model call for every evaluated unit and criterion. A run containing 10,000 outputs and 20 criteria therefore requires 200,000 calls. Latency, token usage, rate limits, and retries all grow with that multiplication.
A classifier bound by a schema
Jev, a hosted product from TypeSafe, accepts text or JSON state and answers predefined questions. Its schema specifies the permitted output type, such as a label from a list, a rubric score, or a yes/no probability. The system also returns a confidence score.
Skipping token-by-token generation reduces inference work and prevents malformed labels or invented options. The schema guarantees a valid output shape, while semantic accuracy still depends on selecting the correct allowed value.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
