Databricks' Genie Beats Rival AI Agents on SQL Tasks With 91% Accuracy

Databricks and Turing built a benchmark showing Genie with its Ontology layer beats two frontier agents 91% to 45-58% on enterprise data tasks.

·
·
·
Read5 min
TypeNews
  • Databricks and Turing built an enterprise data-agent benchmark testing notebooks, dashboards, and saved SQL, not just tables.
  • Genie passed 91% of runs versus 45% and 58% for two unnamed frontier agents.
  • Genie used under half the tokens (432K vs 889K and 1.05M) and ran 3-4x faster.
  • Example task: table shows 0.31 p/kWh but correct FY24/25 planning rate is 0.52 p/kWh, documented only in a notebook.
  • Genie Ontology extracts business context into a retrievable graph linked to source assets.
  • Advantage shrinks on fresh workspaces or pure schema-lookup questions; benchmark is request-only via Databricks account teams.

Databricks has published benchmark results comparing its Genie data agent with two unnamed frontier agents in a deliberately cluttered enterprise workspace. Genie completed 91% of tasks correctly, about 1.6 times the 58% score of the stronger rival. The evaluation tests whether agents can find business rules stored across notebooks, dashboards, documents, and saved SQL, then use that context to produce answers that match the company’s definitions.

Context hidden across a workspace

Conventional text-to-SQL benchmarks usually provide a fixed set of tables, schemas, and pipeline code. Agents perform similarly under those conditions, according to Databricks. The company worked with Turing to build an evaluation around the broader collection of assets that analysts and data engineers use in production.

The fictional test environment represents a UK energy supplier called Calderwick Energy. It contains hundreds of tables, including one with roughly 120 million rows of half-hourly meter data, plus notebooks, scheduled jobs, dashboards, saved queries, source code, metrics, documents, and existing Genie agents.

Workspace clutter forms part of the test. Copied notebooks, abandoned analyses, and stale dashboards force agents to identify the relevant source rather than accept the first plausible artifact they retrieve.

Genie leads on accuracy and efficiency

Each agent attempted every task five times against the same live Databricks workspace. Databricks measured task accuracy, token consumption, and wall-clock time.

Agent Tasks completed correctly Average tokens per run Relative runtime
Genie 91% About 432,000 About 100 seconds
Frontier Agent A 45% About 889,000 About three to four times longer
Frontier Agent B 58% About 1.05 million About three to four times longer
Bar chart showing Genie at 91% accuracy, Agent A at 45%, and Agent B at 58%
Databricks reports that Genie completed 91% of benchmark tasks correctly.

On Databricks’ figures, Genie used about half as many tokens as Agent A and 41% as many as Agent B. Lower token consumption can reduce inference costs, although the dollar difference depends on each system’s underlying models and pricing.

Chart comparing average token consumption for Genie and two frontier agents
Genie consumed fewer tokens per run than either unnamed competitor.

How 0.31 became the wrong answer

One task asked for Calderwick’s Commercial Contracts for Difference, or CfD, planning rate for fiscal year 2024/25. The bill cost stack table contains an apparently relevant cfd_rate_p value of 0.31 pence per kilowatt-hour. A valid query selecting that field would still produce the wrong business answer.

The notebook that builds the mart explains that the Commercial team sets planning rates annually. It also records that the mart uses fiscal year 2025/26 assumptions, including for some earlier billing periods. A separate “Planning rates by year” section preserves the historical value of 0.52 pence per kilowatt-hour for 2024/25.

Databricks reports that Genie returned 0.52 in all five runs. Neither rival consistently found the historical rate, and one returned 0.31 every time. The failure came from applying a current table value to a historical question without consulting the notebook that defined the calculation.

Genie Ontology snippet identifying the notebook source for historical planning rates
The retrieved ontology snippet points to the notebook containing the historical planning-rate definition.

Genie Ontology indexes enterprise context before a question arrives. Its extraction layer builds a graph of snippets connecting metric definitions, calculation rules, and source assets such as notebooks, dashboards, and saved queries. Its retrieval layer selects relevant snippets for each request and includes pointers to their sources.

Precomputing those connections reduces the material Genie must inspect during a run. The agent can retrieve a focused set of definitions and source references instead of repeatedly loading metadata for hundreds of tables or rereading entire notebooks.

Databricks describes the extraction and retrieval architecture in its product documentation. Genie Ontology is generally available.

The claim has firm boundaries

The reported advantage depends on several conditions that limit how broadly the results apply:

  • The workspace is mature. Relevant data is governed through Unity Catalog, Databricks’ data and access catalog, while business knowledge has accumulated across analytical assets.
  • Simple questions reduce the gap. New deployments and tasks answerable directly from tables or schemas provide less extra context for the ontology to retrieve.
  • The rivals lacked ontology access. Their Databricks toolbelt excluded the Genie MCP server, which would have exposed ontology features through the Model Context Protocol used by agents to call external tools.
  • The scores cover complete systems. Results include each agent’s permitted retrieval tools and orchestration, so they cannot isolate the quality of the underlying language models.
  • Independent reproduction is difficult. Databricks does not identify Agent A or Agent B, and developers must request the benchmark through a Databricks account team rather than download it publicly.

Implications for agent design

The benchmark supports Databricks’ argument that retrieval architecture increasingly determines data-agent performance. A model can generate syntactically valid SQL while missing historical assumptions, internal metric definitions, or calculation rules stored outside the queried tables.

Developers building similar systems can apply three practical lessons:

  • Index tables, notebooks, dashboards, saved queries, documents, and metric definitions in a shared retrieval layer.
  • Return source references with extracted business rules so users and downstream systems can verify the answer.
  • Test agents on historical definitions, conflicting artifacts, and stale workspace content alongside standard text-to-SQL tasks.

Established Databricks deployments with years of governed workspace assets fit the benchmark’s strongest case for Genie. Greenfield deployments and schema-bound workloads offer less opportunity for ontology retrieval to change the result. The published numbers therefore measure the value of accumulated business context and the system that retrieves it, rather than language-model capability in isolation.

Trending
  • No trending articles

Comments

avatar

Next Reads