Google Research's GeminiSQL-2 Tops the Hardest SQL Benchmark Without Agentic Tricks
Google Research's GeminiSQL-2 tops the BIRD benchmark for text-to-SQL, using a fine-tuned Gemini 3.1 Pro to turn plain English into execution-ready queries
- GeminiSQL-2 is Google Research's new text-to-SQL system powered by Gemini 3.1 Pro, claiming state-of-the-art on the BIRD benchmark.
- It scored 76.13% execution accuracy on BIRD's Single Trained Model Track, beating all other single-model solutions (human experts score 92.96%).
- Built with a 3-stage recipe: data filtering, multitask fine-tuning, and self-consistency at inference -- no heavy agentic pipelines needed.
- Improvements are live in AlloyDB AI, BigQuery conversational analytics, and Google Code Assist for SQL generation.
- The fine-tuning recipe uses the Vertex AI Supervised Tuning API, meaning teams can replicate it on their own schemas.
- Key caveat: BIRD has known annotation errors, and benchmark accuracy still does not fully predict production reliability on complex enterprise databases.
Converting plain English into working SQL has been a stubborn problem for AI. Queries that look right often fail when you actually run them, especially against real-world databases with messy schemas, ambiguous column names, and domain-specific business logic baked in. Google Research just announced GeminiSQL-2, a new text-to-SQL capability powered by Gemini 3.1 Pro that claims state-of-the-art results on the hardest standard benchmark in the field.
The benchmark that actually matters
The gold standard for evaluating text-to-SQL systems is BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation). BIRD is an industry standard for testing text-to-SQL solutions, spanning over 12,500 unique question-SQL pairs from 95 databases with a total size of 33 GB. What makes BIRD particularly hard is that it does not just check whether a query looks syntactically correct. It moves beyond simple, single-table queries to cover real-world challenges like reasoning over very large schemas, dealing with ambiguous values, and incorporating external business knowledge. Crucially, BIRD measures execution-verified accuracy -- meaning the query must actually run and return the right result.
On BIRD, human experts achieve an execution accuracy of 92.96%, whereas even the top-performing methods lag considerably, with the top approaches around 75% on the test set. That gap is where GeminiSQL-2 is pushing.
The score and what it means
Google Cloud scored a new state-of-the-art result on the BIRD benchmark's Single Trained Model Track, scoring 76.13, ahead of any other single-model solution. The Single Trained Model Track is the most meaningful category to win. It is designed to measure the raw, intrinsic capability of the model itself, restricting the use of complex preprocessing, retrieval, or agentic frameworks often used to boost model accuracy. In other words, no ensembles, no retrieval-augmented generation pipelines, no re-rankers -- just the model.
For context, here is where GeminiSQL-2 sits relative to other approaches on BIRD:
| System | BIRD Execution Accuracy | Track |
|---|---|---|
| Human experts | 92.96% | -- |
| MARS-SQL (7B RL-trained) | 77.84% | Open agentic |
| GeminiSQL-2 (Gemini 3.1 Pro) | 76.13% | Single trained model |
| CHASE-SQL + Gemini | 74.90% | Closed-source agentic |
| Gemini 2.0 Flash (baseline) | ~62% | Zero-shot |
How they built it
The result did not come from just plugging in a more powerful base model. The team developed a three-stage specialization recipe applied on top of Gemini 3.1 Pro. A combination of clean data, multi-task learning, and efficient self-consistency allowed them to take Gemini 3.1 Pro and build a specialist that achieved the top-ranking score on the BIRD single-model benchmark.
The three phases were:
- Rigorous data filtering. The team ran a two-stage pipeline: first, execution-based validation that discarded any training query that failed or returned an empty result; then LLM-based validation where multiple models acted as judges to verify semantic alignment between the question and the SQL. The result was a smaller but much cleaner training set.
- Multitask supervised fine-tuning (SFT). Rather than training the model purely on question-to-SQL pairs, they trained it on several related tasks simultaneously -- schema linking, query planning, and self-correction -- to build deeper SQL reasoning. This proves the core thesis: by investing in high-quality data and instruction tuning, you can build a single model that is powerful enough to be production-ready without requiring a heavy, high-latency inference framework.
- Self-consistency at inference. Instead of generating one query, the model generates several candidates (between 1 and 7), executes all of them, clusters the results, and returns a query from the most common result cluster. They found their sweet spot in the "Few" (1-7 candidates) category, which gave the final critical boost in execution accuracy that pushed the model to the top of the leaderboard.
Self-consistency is the key inference trick here. The intuition is simple: if the model independently arrives at the same answer via different reasoning paths, that answer is far more likely to be correct. It is a lightweight alternative to full agentic ensembles, and it keeps latency manageable.
Where it shows up in Google's products
This is not a pure research result. The improvements are flowing directly into Google Cloud's data services. Three products benefit immediately:
- AlloyDB AI natural language queries: Lets end users query live operational databases in plain English. Companies like Hughes (an EchoStar subsidiary) already use this for call analytics.
- BigQuery conversational analytics: Business analysts can explore large historical datasets, run reports, and extract business intelligence through a multi-turn chat interface without writing SQL.
- Google Code Assist: Developers have long relied on Google Code Assist for code generation, aiding application development with databases across Spanner, AlloyDB, and Cloud SQL Studio. With more accurate NL2SQL, developers can now use AI coding assistance to generate SQL code too.
The official Google Cloud blog post details how these capabilities are being deployed across the data stack. The fine-tuning itself was done using the Supervised Tuning API for Gemini on Vertex AI, which means the same recipe is available to teams who want to fine-tune Gemini on their own database schemas.
What this does not solve
The BIRD benchmark, while rigorous, has known limitations. Prior work has reported annotation errors in 32% of problems in the BIRD mini development set and 49% of financial domain problems, with the latter resulting in up to 17.6% underestimation of performance. So the real-world gap between benchmark scores and production reliability is real.
Benchmarks report impressive numbers, often 85-90% execution accuracy. But a growing body of research and enterprise experience reveals that academic success does not translate to enterprise reliability. Production databases have hundreds of tables, evolving schemas, and business rules that no benchmark fully captures. GeminiSQL-2 is a strong foundation, but teams deploying it against complex proprietary schemas will still need to invest in schema documentation, metadata enrichment, and query validation layers.
The bigger shift
The real signal here is architectural. Google's approach demonstrates that a single fine-tuned model, without complex multi-agent scaffolding, can now compete with or beat ensemble systems that chain multiple models together. That matters for production: simpler inference graphs mean lower latency, lower cost, and fewer failure modes to debug.
It also reframes what fine-tuning is for. The lesson from GeminiSQL-2 is not just "use a bigger model." It is that data quality and task decomposition during training -- teaching the model to plan, link schemas, and self-correct -- can extract dramatically more capability from a model you already have access to. That is a pattern worth applying well beyond SQL generation.