Cohere Catches AI Coding Agents Ignoring Answers Sitting Right in Front of Them
A new paper introduces solution injection, showing agents find task solutions in 90 percent of runs but exploit them in under 7 percent.
PRO- New paper defines environmental curiosity as the missing capability in current LLM agents.
- Solution injection method places complete answers into Terminal-Bench, SWE-Bench Verified, and AppWorld environments.
- Agents discover injected solutions 79 to 81 percent on Terminal-Bench but interact only 37 to 50 percent.
- On AppWorld, gpt-oss-120b sees the solution API in 97.5 percent of runs and calls it 0.53 percent.
- Bash-only scaffolds, higher reasoning budgets, and explicit exploration prompts all improve interaction and task performance.
- Narrow in-distribution fine-tuning suppresses curiosity and degrades
pass@kscaling on original benchmarks.
A five-author team behind the new paper Agents Explore but Agents Ignore has put a name to a failure mode many people building coding agents have seen but not measured: even when the answer sits in front of the model, it usually walks past it. The work is from Leon Engländer, Sophia Althammer, Ahmet Üstün, and Tom Sherborne of Cohere, together with Matthias Gallé of Poolside.
Their central claim, backed by experiments on three popular agent benchmarks, is that current LLM agents behave as open-loop sequence generators. They use the environment to fetch expected information, but rarely to revise a strategy or exploit unexpected stimuli.
Leaving the answer on the desk
To measure this, the authors introduce a method they call solution injection: placing the complete task solution directly within the agent's environment so that following it guarantees success and typical agent actions would surface it.
In practice, that means dropping a solution.sh script into the working directory for Terminal-Bench and SWE-Bench Verified, or adding a documented cli solution endpoint whose docs state it "returns the complete solution to this task" for AppWorld. Two new metrics sit alongside the usual pass@k:
- discovery@k: did any of k attempts surface the injected solution in the agent's context?
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
