Skip to main content
Back

KU Leuven Catches LLMs Confidently Failing Basic Cause-and-Effect Reasoning

A new benchmark tests whether nine leading models can trace the hidden mechanisms between cause and effect, and the results expose reasoning shortcuts.

Pro

  • Nine LLMs including o1, GPT-4o, DeepSeek R1 tested on implicit causal chain discovery for climate arguments.
  • Roughly 50% of reversed cause-effect pairs still labeled causal, exposing weak directional understanding.
  • Active-to-passive voice flips change causal judgments by up to 51%, revealing position-heuristic pattern matching.
  • Humans confirmed 27 of 36 o1-generated chains as valid, disagreeing with LLM self-assessment half the time.
  • More chains per pair correlates negatively with chain length across every tested model.
  • Code, prompts, and generated chains released on GitHub as a reproducible baseline.

A team from KU Leuven and Università della Svizzera italiana just put nine large language models through a diagnostic gauntlet designed to answer a deceptively simple question: when a model says X causes Y, can it actually explain the intermediate steps that get you from one to the other? The paper introduces a task called implicit causal chain discovery, and the findings suggest that current LLMs are confident storytellers about cause and effect but shaky mechanistic reasoners.

Confident claims, missing middles

Causal claims in everyday discourse tend to skip the middle. Speakers omit the underlying mechanism and trust listeners to fill it in. That works for humans because we mentally break a claim like "climate change causes flooding" into intermediate steps such as climate change to excessive rainfall to flooding.

The authors argue this matters most in polarized argumentation. Skeptic and believer groups attribute and accept causality in climate events differently, and each side maintains distinct mechanistic views that reinforce in-group arguments and refute out-group ones. For an LLM to help evaluate arguments, detect fallacies, or generate counterarguments, it needs to surface those hidden mechanisms rather than parrot a surface-level cause-effect pair.

How the task works

Given a cause-effect pair such as CO2 emissions to ocean acidification, the model has to output N different causal chains, each of the form E_c to E_1 to E_2 to ... to E_f. The authors deliberately let the model decide how many chains to produce and how long each should be, because forcing a fixed count would bias the evaluation. They use zero-shot prompting with no examples, no chain-of-thought scaffolding, and no retrieval augmentation, since the goal is to probe what causal knowledge is baked into the parameters themselves.