Iowa State Teaches AI to Reason Over Databases by Showing It Wrong Answers
A new paper from Iowa State and Thoughtworks shows that teaching LLMs what NOT to do — using synthetically generated wrong reasoning paths — dramatically boosts multi-table question answering accuracy.
- Core result: CPO training on synthetic contrastive reasoning traces improves multi-table Q&A by 9.7–16.3% on average across three open-weight LLMs, with up to 21 percentage points on MMQA.
- Key method: GPT-4o generates correct step-by-step reasoning traces; Gemini 2.0 Flash generates plausible-but-wrong versions; models are fine-tuned on the pairs using Contrastive Preference Optimization (CPO).
- Heterogeneous generators matter: Using different LLMs for positive and negative traces consistently outperforms using the same model for both, because it creates stronger distributional separation.
- DPO fails here: Standard DPO training actually degrades performance on structured multi-table reasoning; CPO's reference-free design and behavior-cloning regularizer stabilize training on long reasoning chains.
- Generalizes to 3-table queries: Models trained only on two-table data transfer their gains to harder three-table examples, suggesting learned reasoning patterns rather than dataset memorization.
- Code and dataset: The authors plan to release both publicly; open-weight Qwen-3 30B can substitute for proprietary APIs with competitive (though slightly lower) performance.
Getting a language model to answer questions that span multiple database tables is genuinely hard. It's not just about reading data , the model has to figure out which tables are relevant, understand how they link together through shared keys, and then chain several reasoning steps without dropping a constraint along the way. Most existing training datasets for this problem give models only the question and the final answer, leaving the model to figure out the reasoning path on its own. A new paper from researchers at Iowa State University and Thoughtworks argues that this missing middle layer , the step-by-step reasoning trace , is exactly what's holding models back, and that showing models wrong traces is just as important as showing them right ones.
The gap nobody was filling
Multi-table question answering requires models to retrieve relevant evidence, link schemas, and perform compositional reasoning across relational tables , but existing resources typically provide questions and final answers without reasoning supervision that explains how answers are derived. This matters because multi-table reasoning is qualitatively different from single-table work. A model might need to identify a foreign key relationship between two tables, filter one table based on a constraint, join the result with another, and only then compute an aggregate. Miss one constraint and the answer is wrong , but the model gets no signal about where it went wrong.
The paper introduces a pipeline to fix this by generating synthetic contrastive reasoning traces: paired examples of correct step-by-step reasoning paths and plausible-but-wrong ones. The authors construct a synthetic contrastive reasoning-trace dataset for MMQA by generating validated positive traces and plausible negative traces with heterogeneous LLMs, then use the resulting preference pairs to fine-tune open-weight LLMs with Contrastive Preference Optimization (CPO).
How the pipeline works
The approach has three distinct stages, each with a different model playing a different role:
- Generate positive traces. GPT-4o is prompted to produce step-by-step reasoning traces for each question-table pair. Every step must explicitly name which table it references, follow a structured markdown format, and use short identifiers rather than copying full rows. A trace is only accepted as "positive" if a Gemini 2.0 Flash judge confirms its final answer matches the ground truth , 58.33% of generated traces pass this filter.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
