EvolveScaler tests long-context models against changing facts

Long-context benchmarks usually ask models to retrieve a fact from a large block of static text. Researchers at Tencent Hunyuan and Renmin University have introduced EvolveScaler to test a harder task: tracking facts that are corrected, revoked, or replaced as a conversation unfolds. The framework defines each scenario in executable code, renders it as dialogue, and calculates the correct answer by replaying the underlying events.

When old facts stop counting

The researchers call this setting information evolution. A long context may describe an evolving process in which later events change the validity or effect of earlier records. In a shared-expense chat, someone might correct an amount two days later. In a warehouse log, a scan might be retracted after a stock count. Answering a question about the current state requires reconstructing the history and applying each update in order.

Text-first data generation makes these scenarios difficult to verify. One model may write a long dialogue while another infers the expected answer, leaving state transitions and answer logic implicit. As traces grow, records can conflict, difficulty becomes hard to control, and some questions may lack enough information for a definite answer.

Code decides the answer

EvolveScaler starts with a human-authored operational specification for each workflow. The specification acts as a typed state machine, a formal model that defines the available records, the rules for updating them, and the conditions under which each record remains valid. The generation pipeline then follows five steps:

1. Humans define state transitions, validity rules, difficulty controls, and executable answer logic.
2. A language model converts the specification into a self-contained simulator.
3. Automated checks validate the simulator and its outputs.
4. The simulator generates natural-language, multi-turn event histories.
5. Deterministic replay calculates reference answers and small, checkable answer components.

Under this contract, the language model writes the simulator while validated code computes the labels. The approach provides several practical properties:

• Validated transitions enforce the workflow’s consistency rules.
• Reference answers come from deterministic replay.
• Length, invalid-record density, and counterfactual depth remain configurable parameters.
• Each sample can be checked for answerability before release.

Rendered dialogues expose local changes such as stock +2 without revealing running totals. A model must reconstruct the state from the full sequence instead of copying a cumulative value from the text.

From seven events to 1,200

The published corpus spans 12 workflow themes with frequent updates and revisions, including shared expenses, warehouse shifts, sports play-by-play, and e-commerce returns. Its main components are summarized below.

The 159 operators support different questions over the same event history. A single trace can produce a net-total query, a ranking, a temporal slice, or a counterfactual such as “What would the result be if event 47 had never occurred?” Reusing the trace in this way tests distinct access and reasoning patterns while holding the underlying evidence constant.

Long traces expose a steep drop

The researchers evaluated 14 frontier and open models across all five difficulty levels. Their primary metric, avg@5, measures mean success over five attempts. Performance stayed relatively close on short traces and diverged sharply on the longest ones.

Error labels reveal where performance breaks down. Net-change aggregation causes roughly half of the classified failures: models often retrieve the relevant records and then combine their signed effects incorrectly. Aggregation, retrieval, and counting together account for about two-thirds of failures. The strongest overall system had the smallest share of aggregation errors and the largest share of ranking errors, indicating that leading models have different bottlenecks.

Six thousand samples transfer

The authors fine-tuned an internal model identified as A3B on 6,000 EvolveScaler samples, then tested it on eight independently constructed out-of-distribution benchmarks. The trained checkpoint outperformed the base model on all eight evaluations, with an average gain of 5.25 percentage points.

Gains across separately built evaluations suggest that training improved the broader skill of maintaining valid state through a changing history. The result is more useful for practitioners than an improvement confined to EvolveScaler’s own question formats, though the reported experiment covers one internal model and one training configuration.

Where the benchmark fits

EvolveScaler targets systems that must maintain structured state across long, mutable histories. Likely applications include:

• Agents operating over multi-day chats, support tickets, or logs containing edits and retractions.
• Assistants that maintain ledgers, inventories, calendars, or other structured records across turns.
• Evaluation pipelines that need code-verified labels for synthetic long-context data.
• Training systems that must interpret backfilled, cancelled, or counterfactual events.

Coverage remains focused on structured workflows with explicit state changes. The benchmark offers limited evidence about open-ended conversation, general mathematical reasoning, or products built on immutable histories.

Paper, demo, and code

The paper, an interactive project page, and the GitHub repository are publicly available. The project page includes a warehouse-log walkthrough that shows how local events accumulate into the final state.

The reusable method is a specify, simulate, validate, and render pipeline for generating long-context data with executable ground truth. Developers could adapt the same design to code-review histories, multi-agent negotiations, audit trails, or other domains where later events alter the meaning of earlier records.