Vals AI's Vals Index v2.2 Reshuffles Leaders With Stricter Grading

Vals AI released Vals Index v2.2, its leaderboard for models acting as agents on finance, coding, legal, and tax tasks. Claude Sonnet 5.5 now leads, while stricter judges, revised hidden tests, and an all-pass tax rubric have lowered many scores. Because the evaluation changed alongside the rankings, comparisons with earlier versions can mix model progress with scoring changes.

Three models sit within 1.61 points

The component leaders split by domain, revealing different strengths beneath the aggregate scores:

• Claude Opus 5.5 leads Finance Agent v2.1 at 67.85%, Terminal-Bench 4.0 at 65.15%, and Tax Agent Bench at 45.57%.
• Claude Sonnet 5.5 leads Excel Modeling Benchmark v1.1 at 75.90%, Vibe Code Bench v1.1 at 92.39%, and Code Migration v1.1 at 87.37%.
• Gemini 4 Argon leads Legal Research at 54.81% and Harvey’s LAB v1.1 at 27.08%.

Refusal handling changes first place

The published Sonnet and Opus scores include some component tasks completed by a fallback model after the primary provider refused the request. Counting each refusal as a failed task lowers Sonnet by 1.82 percentage points and Opus by 6.02 points. Gemini used no fallbacks and ranks first under that policy.

Deployment policy determines which view is more relevant. Systems that route refused requests to another model resemble the published methodology; systems that treat refusals as hard failures need the adjusted scores.

DeepSeek V4.1 Flash 1017 ranks eighth among open-weight and proprietary models, scoring 61.26% at a reported cost of $0.66 per test. Using published scores, it trails Sonnet by 9.59 percentage points and Opus by 8.92 points. Sonnet costs 28.3 times as much per test at $18.70, while Opus costs 42.6 times as much at $28.12.

Five changes tighten the test

• Finance Agent v2.1 uses a new panel of judge models. Answers that hedge between several possible values now fail, closing a scoring loophole that rewarded lists containing the correct number.
• Excel Modeling Benchmark v1.1 uses GPT-6 Luna as its judge. The lower judging cost allows Vals to run more repeated evaluations for each task.
• Code Migration v1.1 runs every model through the Pi agent harness and uses revised hidden tests. The shared harness improves consistency, while the new tests reduce advantages from familiarity with older test patterns.
• Harvey’s LAB v1.1 gives judges access to tracked changes in redlined documents. That allows the benchmark to score the edits themselves during long legal drafting tasks.
• Tax Agent Bench now uses all-pass accuracy. A response earns credit only when every rubric requirement passes, with no partial credit.

Changes to judges, test cases, execution harnesses, and scoring rules establish a new measurement baseline. Model comparisons within v2.2 remain consistent, while claims about improvement across index versions require task-level reruns under the same rubric.

Finance and coding account for 88.9%

Vals combines the four sectors according to their estimated shares of U.S. gross domestic product, using Bureau of Economic Analysis value-added-by-industry data. Each sector score first averages its component benchmarks, then receives its GDP-derived weight:

Finance = AVG(FinanceAgentV2_1, EMBV1_1)
Coding = AVG(TBench4, VibeCodeBenchV1_1, CodeMigrationV1_1)
Legal  = AVG(LegalResearch, HLABV1_1)
Tax    = TaxAgentBench

Vals_Index = (8.0*Finance + 5.6*Coding + 1.2*Legal + 0.5*Tax) / 15.3

A model’s finance and coding performance drives almost nine-tenths of its final score. Strong legal or tax results have limited influence on the aggregate, so domain-specific buyers gain more information from the component leaderboards.

Read the task scores before buying

The index supports initial model screening through private tasks drawn from recognizable workflows: multi-step financial analysis, spreadsheet construction, COBOL migration, legal drafting, and cited corporate tax research. Private items reduce contamination risk, while reported accuracy, cost per test, and elapsed runtime expose practical deployment tradeoffs.

Limits that can change a buying decision

• Legal and tax receive small weights. A law firm or tax practice may find the aggregate ranking poorly aligned with its workload.
• The coding weight uses the information sector as a proxy. That mapping can give consumer software more influence than internal tooling receives within a particular organization.
• Private tasks restrict external auditing. Teams can inspect the methodology and reported outcomes, though they cannot independently review every test item.
• Aggregate scores hide failure modes. Refusal rates, weak task categories, latency spikes, and harness compatibility can matter more than a one-point difference in the index.

Stricter grading raises the reliability bar

Unambiguous financial answers, visible redline edits, and complete tax rubrics resemble production acceptance criteria in regulated workflows. Lower v2.2 scores can therefore reflect harder evaluation alongside model behavior. Future gains measured within this baseline will provide a cleaner reliability signal, while cross-version claims still require controlled reruns.