TableMaster Pushes GPT-4o-mini from 59% to 78% on Complex Table Queries
A three-stage framework dissects why language models fail on tables, then patches each failure with a specific fix, hitting 78% on WikiTQ with GPT-4o-mini.
PRO- TableMaster identifies four distinct failure modes in LLM table understanding and patches each with a targeted technique.
- Hits 78.13% on WikiTQ with GPT-4o-mini, 13.4 points above prior best.
- Three-stage pipeline: structure extraction, content verbalization, then adaptive textual or symbolic reasoning.
- Uses roughly 4x fewer prompt tokens than Chain-of-Table while scoring higher on TabFact.
- Lifts GPT-4o-mini on FinQA numerical reasoning from 50.7% to 66.4% accuracy.
- Full paper and prompts published, though no dedicated public code repository was released.
Language models can write poems and debug code, but ask one to count how many Belgian riders won a race in a Wikipedia table and it will often miscount rows or hallucinate a total. A new framework called TableMaster treats table understanding as four separate failure modes, each with its own patch. On the WikiTQ benchmark, the method pushes GPT-4o-mini from 59.5% to 78.1% accuracy without any fine-tuning.
Why tables break language models
Tables look simple, but they violate almost every assumption baked into pretraining corpora. Tabular data is structured, dense, concise, and numerical, roughly the opposite of the flowing prose LLMs learn from. The authors ran a diagnostic sweep and pinned down four specific reasons models trip up.
- Locating target data. Performance declines as table size increases, and for weaker LMs the drop is more pronounced. Models get lost when the answer sits in row 47 of 100.
- Sparse semantics. A cell that just says "1998" means nothing without the header and neighboring context. Semantic deficiency in tables makes it hard for LMs to process the data effectively.
- Arithmetic in prose. Textual reasoning suffers a 20.1% decline on calculation-heavy questions, whereas enhanced symbolic reasoning experiences a more moderate drop of 7.6%.
- Rigid code generation. When asked to write Python, models often regurgitate memorized snippets instead of actually reading the table. On the same content in a noisy format, symbolic reasoning suffers a 31.8% performance drop compared with 20.5% for textual reasoning.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
