Skip to main content
Back

ORCA Cuts AI Causal Analysis Errors by 6x on Messy Real Databases

A multi-agent system with human checkpoints orchestrates causal analysis across relational databases, cutting data exploration time by 76 percent and dramatically improving effect estimation accuracy.

Pro

Paper, Full Access
  • ORCA is a multi-agent framework coordinating three specialist agents for causal analysis on relational databases.
  • Human checkpoints prevent early data-exploration errors from silently invalidating downstream ATE estimates.
  • Cut data exploration time 76% and improved CI coverage from 8% to 50% versus GPT-4o-mini baseline.
  • Introduces REEF, a semi-synthetic relational e-commerce benchmark with a known Structural Causal Model.
  • Discovery uses diagnostics-driven algorithm selection and ensembles PC, FCI, GES, LiNGAM, NOTEARS outputs.
  • Open-source implementation available at anonymous.4open.science/r/ORCA-E70F.

Running causal analysis on a real production database is a punishing exercise. Variables sit scattered across dozens of tables, joins reshape the unit of analysis in subtle ways, and one wrong aggregation early in the pipeline can silently invalidate every estimate that follows. A new framework called ORCA argues that this is a coordination problem rather than an automation problem, and that a well-designed multi-agent system with human checkpoints can dramatically improve both the speed and reliability of end-to-end causal work.

The consistency problem nobody was solving

The paper starts from a concrete observation. When the authors watched twelve data-literate participants perform causal tasks with a GPT-4o-mini coding assistant, participants spent 77% of their total analysis time on data exploration alone, and repeated exploration made it difficult to track whether earlier intermediate artifacts remained valid. The downstream damage was severe: only 8.3% of final ATE estimates fell within the ground-truth confidence interval.

The authors call the underlying issue cross-stage consistency. Roughly, the join and aggregation choices made during exploration have to match the variable definitions handed to causal discovery, the discovered graph has to encode only assumptions that actually hold on the constructed dataset, and the identification strategy used for effect estimation has to remain valid under those same assumptions. Existing LLM tools automate individual steps while happily letting users break these invariants.