ReViSQL-K2.6 Beats Human Accuracy on SQL Benchmark Without Agentic Scaffolding

Text-to-SQL has been one of the more embarrassing gaps in modern LLM benchmarks. A new collaboration between Thinking Machines Lab, UIUC, and Bridgewater AIA Labs just closed it, producing the first single model to cross the human accuracy line on the BIRD benchmark without any agentic scaffolding.

The model, ReViSQL-K2.6, is a fine-tune of Kimi-K2.6 trained with reinforcement learning with verifiable rewards (RLVR) on Tinker. Under greedy decoding it scores 91.37% on Arcwise-Plat-SQL at $0.035 per task, and 92.97% at $0.56 per task when voting among 16 samples, the first time a text-to-SQL AI system has crossed the 92.96% human benchmark.

Why SQL has been so stubborn

Humans score 92.96% on BIRD, a realistic benchmark for translating natural-language questions into SQL, but AI has trailed for years. LLM scores on the BIRD leaderboard climbed from just below 70% in 2024 to 82% today, with frontier models like GPT-5.6 Sol Ultra and Claude Fable 5 landing in the mid-80s at a cost prohibitive for high-volume applications. Syntax was never the hard part. The real difficulty is disambiguating fuzzy business questions against schemas that can contain thousands or millions of columns.

The dominant workaround has been elaborate scaffolding. Text-to-SQL pipelines use a schema-linking stage to narrow columns, a generation stage to sample queries, a self-correction stage to repair execution errors, and a selection stage to vote among survivors. Every step is a separate model call, tuned per benchmark, and the resulting systems still lag humans by 11 points.

The training data was lying

The team's first insight was that RLVR's usual failure mode here is not the algorithm but the labels. In RLVR, the scalar reward is the entire learning signal, so mislabeled examples flip the gradient. When they audited 2.5k instances from BIRD Train, the picture was ugly:

• 52.1% of gold SQL queries were incorrect
• 26.2% of the natural-language questions were flawed
• 18.2% of external knowledge entries were wrong
• 61.1% of instances had at least one of these errors

Cleanup ran in multiple stages: an o3 auditor plus a human expert flagged errors, a second expert verified, and disagreements went into conflict resolution loops. The LLM auditor caught mistakes precisely (90.6% precision) but only surfaced 24.5% of what humans caught, which is worth remembering the next time someone offers to "just use an LLM judge" for data cleaning. The resulting dataset, BIRD-Platinum, is public.

Cleaner data alone did most of the heavy lifting. Kimi-K2.6 fine-tuned on BIRD-Platinum hit 88.55% on Arcwise-Plat-SQL, beating GPT-5.6 Sol Ultra (86.75%), Claude Fable 5 (84.94%), and every open-weight fine-tune tested. To rule out benchmark overfitting, the team also fine-tuned Qwen3-235B-A22B and evaluated on harder benchmarks. Training on BIRD-Platinum improved accuracy by 16% on Arcwise-Plat-SQL, 12% on Spider2-SQLite, and 14% on Spider2-Snow versus training on the original BIRD Train.

Fixing two subtle reward bugs

Even with clean data, a 4-point gap to humans remained. The team traced it to two divergences between the standard result-based reward and the actual behavior they wanted.

Execution match is not semantic equivalence. The default reward checks whether the generated SQL returns the same result as the gold query on a single database instance. That leaves a lot of room for wrong queries to score correct. Using VeriEQL, a bounded verification solver that adds under 0.1% to training compute, they found that 32.8% of positive rewards were going to queries that were not actually equivalent to the correct one. Nearly one reward in three was reinforcing the wrong pattern. They downweighted rewards for queries that pass execution but fail VeriEQL.

Outcome rewards ignore provided context. BIRD problems ship with external knowledge hints (for example, "sodium-free refers to sodium = 0"). A model that guesses "sodium < 5" from its pretraining prior can still land on the same result set and get rewarded. In pilot analysis, 24.2% of failures came from the model ignoring supplied information. The fix is a rule-based process reward: the model must emit a requirements block that translates each knowledge entry into query constraints, plus a verification block that audits its own SQL against those constraints, with penalties for non-compliance.

The training recipe

The full setup is reproducible on the Tinker code. Notable settings:

• Base model: moonshotai/Kimi-K2.6
• RL objective: CISPO, batch size 64, group size 16
• LoRA rank 32, learning rate 5e-5
• 32,768 max input tokens, up to 5 model-environment interaction turns
• VeriEQL penalty of 0.2 for execution-match-but-not-equivalent queries
• Process reward penalty of 0.1 for skipping the external-knowledge analysis

Rethinking scaffolds versus training

The results reframe the tradeoff between orchestration and model quality. The greedy-decoding version beats OpenSearch, the strongest prior open-source pipeline, by 8.4 points at 37% lower cost, largely because there is no auxiliary scaffolding around the model. Self-consistency at 16 samples barely counts as a scaffold either, since it just samples from one prompt and takes the majority vote, with no extra prompted stages.

The practical takeaways for anyone building domain-specific models:

1. If you are doing RLVR on a public benchmark, assume the labels are wrong until proven otherwise. A majority of BIRD Train's gold queries were incorrect.
2. LLM-as-auditor is a starting point, not the whole pipeline. Recall was under 25% versus expert reviewers.
3. Result-based rewards can silently reinforce wrong reasoning. If your task has a symbolic verifier available (like VeriEQL for SQL, or unit tests for code), use it as a secondary reward channel.
4. When a prompt includes structured context the model must use, add a process reward that forces the model to explicitly reference and check that context.

There is also a broader point about where task expertise should live. Scaffolds encode expert knowledge in orchestration code and prompts. This work argues that the same knowledge, applied instead to data curation and reward shaping, produces a model that outperforms the pipeline at a fraction of the cost and generalizes to harder benchmarks the training data never touched. For anyone maintaining a stack of glue code around a frozen frontier model to hit accuracy targets, that is worth a hard look.