CMU's CLARITI Beats GPT-5 on Coding Tasks While Asking Half as Many Questions
An 8B clarification module matches GPT-5's fix rate on underspecified GitHub issues while asking 41% fewer questions, using empirically-grounded rewards.
Takeaways
- CLARITI is an 8B Qwen3-based clarification module that matches GPT-5's fix rate on underspecified SWE-Bench issues.
- Achieves 36.8% task success versus GPT-5's 35.6%, while asking 3.0 versus 5.1 questions on average.
- Uses a four-stage GRPO reward pipeline: non-redundancy, diversity, answerability, and SHAP-weighted task relevance.
- Shapley analysis shows Error Information is the highest-impact category, though Expected Behavior is more often missing.
- Ablations show removing the answerability reward collapses performance back to the no-clarification baseline.
- Code and data available at github.com/sani903/Teaching-Effective-Clarification.
Coding agents fail a lot when instructions are vague, and the obvious fix of asking clarifying questions has its own trap: pile on too many and users tune out; ask the wrong ones and you burn tokens without unblocking the task. A new paper from Carnegie Mellon teaches a small model exactly which questions matter. The result is an 8B parameter clarification module called CLARITI that keeps pace with GPT-5 on underspecified GitHub issues while asking roughly half as many questions.
The framing is refreshingly practical. The authors quantify which types of information most affect task success and which questions elicit useful responses from simulated users, then bake those findings directly into reinforcement learning rewards. Two properties emerge as the pillars of good clarification: task relevance (does the missing information actually predict success?) and user answerability (can the person who filed the issue realistically supply it?).
Why software engineering is the right testbed
SWE issues make a natural stress test for clarification because success is objectively measurable via test suites, and real bug reports routinely omit critical details. Converting fully specified issues into underspecified variants drops agent performance from 43.8% to 23.7% across 700 evaluated instances, confirming that missing information substantially reduces task success. The open question is how to clarify efficiently.
The team hand-coded 112 highly underspecified SWE-Bench Verified issues and derived six categories of missing information: error information, expected behavior, implementation details, external references, reproduction steps, and version/environment. They then ran a Shapley value analysis to see which categories actually predict a successful patch.