Johns Hopkins Finds Maximum AI Reasoning Effort Costs 42x and Solves Less

Many frontier language-model APIs expose a reasoning-effort control with low, medium, high, and sometimes maximum settings. These controls allocate more test-time compute, usually through hidden reasoning tokens, and can sharply increase cost and latency. A new paper from Johns Hopkins, Together AI, and Cornell finds that maximum effort can also discard correct answers produced at lower settings.

The researchers describe reasoning effort as a credence good, an economics term for a purchase whose required quantity remains hard to assess after the transaction. Even after receiving a correct answer, a user lacks evidence that the extra compute was necessary. To estimate that missing counterfactual, the team ran eight proprietary and open-weight models at every available effort level across four benchmarks, measuring accuracy, cost, latency, and agent behavior.

The top setting drops solved cases

Across adjacent effort settings, higher settings gained some benchmark instances and lost others. Aggregate accuracy could rise while individual tasks moved from solved to failed. Each setting generates a separate trajectory, producing overlapping but non-nested sets of solved instances that a single leaderboard score conceals.