Skip to main content
Back

Merkle's BADGER Beats Every SQL Eval Tool by 0.3 Kappa Points

BADGER unifies SQL accuracy and agentic behavior evaluation into one production-grade framework, beating six competing tools with 87.3% balanced accuracy and substantial human agreement.

Paper, Full Access
  • BADGER is a unified evaluation framework from Merkle that combines text-to-SQL accuracy and agentic behavior metrics into one production pipeline.
  • Its Hybrid-EX metric uses an LLM to align column aliases and numeric tolerances before deterministic scoring, fixing a key brittleness in standard execution accuracy.
  • Validated on 150 human-annotated enterprise queries, Hybrid-EX achieves Cohen's kappa of 0.717 and 87.3% balanced accuracy, outperforming all six competing frameworks.
  • The framework introduces Excess Tool Usage as a novel agentic metric, measuring whether an agent calls more tools than necessary to complete a task.
  • BADGER runs entirely within a client's governed data environment and supports pluggable LLM judge backends, making it viable for sensitive enterprise deployments.
  • No public code release has been announced; the paper describes the methodology and validation results in full.

Evaluating AI systems in enterprise settings has always been harder than the leaderboards suggest. BADGER, a new framework developed at Merkle, takes direct aim at this gap by combining text-to-SQL assessment and agentic pipeline evaluation into a single, production-ready system calibrated against real human expert judgment.

The core problem is one that anyone who has deployed a SQL-generating AI in production knows well: the benchmarks that dominate research do not reflect what actually happens at work. The dramatic gap between Spider (86% accuracy) and Spider 2.0 (6%) reveals what benchmarks miss: complex schemas, enterprise-scale databases, multiple SQL dialects, and multi-step reasoning requirements. Meanwhile, the evaluation tooling has not kept pace with the complexity of modern agentic systems that chain together retrieval, reasoning, and SQL generation across multiple steps.

Why existing benchmarks fall short

The text-to-SQL evaluation landscape has a long history. Spider is the most widely used text-to-SQL benchmark, containing 10,181 questions across 200 databases covering 138 domains. But those databases are clean and small. Current text-to-SQL benchmarks test simplified conditions that don't reflect production environments -- Spider 1.0 shows models achieving 85-86% execution accuracy, but these databases contain 10-20 tables with clean schemas and simple business logic, conditions that almost never exist in enterprise settings.

BIRD pushed things forward by adding messier data and external knowledge requirements, but more recent benchmarks including BIRD and its extensions BIRD-CRITIC, BIRD-Interact, and LiveSQLBench have expanded evaluation to complex queries, SQL debugging, and agentic interaction, but still remain focused on text-to-SQL translation. Real enterprise pipelines involve much more: tool orchestration, multi-step reasoning, and queries written in BigQuery, Snowflake, or Redshift dialects with heavy use of CTEs (Common Table Expressions -- reusable named subqueries that make complex SQL more readable but harder to parse automatically).