Jev Cuts Rubric-Grading Costs as LLM Cascades Repeat Its Errors

An arXiv paper by Delip Rao and Chris Callison-Burch of the University of Pennsylvania tests whether rubric evaluations require a generative language model. Across nine benchmark panels, three flash-tier LLM judges cost 29 to 325 times as much as Jev and took 30 to 220 times as long. Accuracy differed significantly in eight of 27 pairwise comparisons, while a simulated confidence cascade improved on the best single judge by no more than 1.5 percentage points with cross-fitted thresholds.

Rubrics multiply the bill

Rubric-based evaluation turns an open-ended response into decisions about individual criteria. A chemistry grader might check whether an answer states a required relation, while an essay grader might assign one of five grammar levels. Benchmarks such as HealthBench use these judgments, and reinforcement-learning systems can use them as reward signals.

Per-criterion graders make one model call for every evaluated unit and criterion. A run containing 10,000 outputs and 20 criteria therefore requires 200,000 calls. Latency, token usage, rate limits, and retries all grow with that multiplication.

A classifier bound by a schema

Jev, a hosted product from TypeSafe, accepts text or JSON state and answers predefined questions. Its schema specifies the permitted output type, such as a label from a list, a rubric score, or a yes/no probability. The system also returns a confidence score.