UESTC's eMoT Hits 100% on a Benchmark That Breaks GPT-4
eMoT gives LLMs an evolving memory of how to reason, pairing it with Python execution to hit 100% on Game of 24 without giant models
Takeaways
- 100% on Game of 24: eMoT hits perfect accuracy on this combinatorial benchmark, surpassing baselines by up to 17.6%, where GPT-4 with CoT scores only 4%.
- Three-module framework: Memory Corrosion (evolving schema store), Symbolic Anchoring (Python execution), and Consistency Refinement (adjudicating disagreements) work together.
- Living memory, not static templates: Reasoning schemas are reinforced when useful and decay when unused, keeping memory compact and current.
- Lightweight backbone wins: Gains come from the framework's reasoning control, not model scale -- a direct challenge to the "just use a bigger model" approach.
- Broad math benchmark coverage: Evaluated on GSM8K, ASDiv, SVAMP, MGSM, and GSM-Hard, showing consistent improvements over Chain-of-Thought baselines.
- No public code yet: The paper is available on arXiv but no code repository has been released, requiring implementation from scratch for replication.
Every time a large language model solves a math problem, it starts from scratch. It has no memory of the last time it saw a similar problem, no record of which reasoning strategies worked, and no way to learn from its own successes. That stateless approach is a core reason why even strong models hallucinate on multi-step arithmetic and drift into logical errors mid-chain. A new paper from researchers at the University of Electronic Science and Technology of China proposes a fix: give the model a living, evolving memory of its own reasoning procedures.
The system is called eMoT (evolving Memory-of-Thought), and it reframes multi-step reasoning not as a one-shot generation task but as a process of retrieving, applying, and continuously refining stored procedural knowledge. The result is a framework that achieves 100% accuracy on the Game of 24 benchmark -- a combinatorial arithmetic puzzle that stumps frontier models -- while running on a comparatively lightweight backbone.
Why reasoning keeps breaking
LLMs achieve impressive performance on multi-step reasoning tasks, but their reliability is persistently hindered by hallucinations and poor numerical computation -- issues that arise fundamentally because standard models treat reasoning as a transient, one-off generation process rather than retaining and refining successful procedural logic. Every inference is isolated. The model cannot accumulate experience about which solution structures tend to work for which problem types.
Chain-of-thought prompting improves reasoning but can increase hallucinations by up to 12% in complex tasks. The original Memory-of-Thought (MoT) approach tried to address this by storing past reasoning chains and retrieving them at inference time. Before the test stage, the LLM thinks over an unlabeled dataset and saves high-confidence reasoning paths in an external memory system; during the test stage, when the LLM encounters a new question, it retrieves relevant thoughts from memory to aid its reasoning -- improving capabilities without updating model parameters. But static memory has a ceiling: stored templates age poorly, clutter accumulates, and there is no mechanism to weed out reasoning patterns that stop being useful.