ChemCoTBench-V2 Catches AI Models Getting Chemistry Right for Wrong Reasons
ChemCoTBench-V2 catches frontier LLMs getting the right answer for the wrong chemical reasons, using deterministic rule-based step verification across 5,620 samples
Takeaways
- New benchmark: ChemCoTBench-V2 evaluates LLM chemistry reasoning at the step level, not just final answers, across 5,620 samples and 18 tasks.
- Core finding: Frontier models frequently produce correct final answers while failing intermediate chemical step checks -- a silent failure mode prior benchmarks missed.
- Key innovation: Uses deterministic chemistry rules (not LLM judges) to verify each reasoning step, making evaluation auditable, reproducible, and cheap to run.
- Three scoring signals: Final-answer correctness, template adherence, and step-wise verifier correctness are reported separately for fine-grained model comparison.
- Task coverage: Molecular understanding, molecule editing, molecular optimization (QED, LogP, protein targets), and reaction prediction across four core categories.
- Open source: Code on GitHub and datasets on Hugging Face under CC BY 4.0.
Getting the right answer in chemistry does not mean you understood the chemistry. That gap -- between a correct final output and a valid reasoning path -- is exactly what a new benchmark called ChemCoTBench-V2 is designed to expose. The paper introduces a rule-verifiable diagnostic framework that checks not just whether a model names the right molecule or product, but whether every intermediate reasoning step it took was chemically sound.
The silent failure mode nobody was measuring
As LLMs get deployed in drug discovery pipelines, materials research, and reaction planning tools, the stakes of silent reasoning failures go up fast. A model that predicts the correct drug candidate through flawed mechanistic logic is not a reliable tool -- it got lucky. Yet most chemistry benchmarks today score only the final answer, which means this failure mode is essentially invisible.
Existing benchmarks often conflate reasoning, knowledge recall, and numerical computation into single exam-style metrics, which obscures whether strong performance stems from structured reasoning or memorized facts. This makes it hard to know whether a model that aces a chemistry eval is actually reasoning about molecular transformations or just pattern-matching to training data.
Chemistry, a discipline fundamental to areas like drug discovery and materials science, still lacks a benchmark that assesses whether LLM reasoning improvements extend to its complex, domain-specific problem-solving needs. ChemCoTBench-V2 is a direct attempt to fill that gap.

What ChemCoTBench-V2 actually measures
The benchmark covers four core chemical task categories, each requiring models to expose their intermediate reasoning in expert-designed structured templates:
- Molecular understanding -- parsing SMILES strings, identifying functional groups, counting substructures
- Molecule editing -- performing substructure additions, deletions, and substitutions on molecular graphs
- Molecular optimization -- improving physicochemical properties like QED (drug-likeness score), LogP (lipophilicity), and solubility, or optimizing for protein targets like DRD2, JNK3, and GSK-3beta