LMSYS's Miles Framework Fixes the Silent Bug Wasting 10x AI Training Compute

Training an agent with reinforcement learning sounds straightforward until you realize the model is not generating one long sequence -- it is making dozens of separate calls to an inference engine, interleaved with tool outputs, harness messages, and retries. Every time you stitch those turns back together for the trainer, you are one subtle bug away from feeding it tokens the model never actually saw. LMSYS's new blog post on their Miles framework tears this problem open and shows exactly how they solved it.

The silent killer in agentic RL

In standard single-turn RL, the trainer evaluates the same token sequence the model generated. Simple. But in agentic settings, a rollout is not a single generation -- it is a chain of model calls, tool outputs, harness messages, and resumed generations. The trainer needs to evaluate the entire trajectory as one contiguous sequence, but that sequence was built across many separate inference calls.

The Token-In-Token-Out (TITO) principle is the invariant that keeps this honest. TITO is a design principle that addresses one critical source of training-inference mismatch: whether the trainer evaluates the exact same token sequence that the inference engine consumed and produced during rollout. Violating it means the trainer is grading tokens based on a context the model never actually saw -- and the model silently drifts off-policy.

Why it costs you 10x compute to ignore this

There are two ways to package a multi-turn trajectory for the RL trainer:

• One sample per turn: Each turn is an independent training sample. Simple, but expensive.
• One sample per task: All turns are glued into one contiguous sequence. Efficient, but only safe if TITO holds.

For a typical SWE-Bench-like task, a trajectory consists of 30-50 turns, which means that to ingest the same amount of information, the one-sample-per-task option only has to spend an order of magnitude less compute compared with the per-turn option. That 10x compute reduction is only achievable if every token in the packed sequence is exactly what the model produced -- otherwise you are training on fabricated context.

The mathematical stakes are just as high. For a training sample to be on-policy, every sampled token should be evaluated by the trainer under the same conditional distribution that produced it during rollout. In transformers, that conditional distribution is entirely dependent on the preceding context of the token. Even a single mismatched token early in the sequence shifts the conditional probability for every token that follows it.

Three ways TITO silently breaks

The LMSYS team identified three common failure modes that are easy to miss in practice:

1. Detokenize-retokenize mismatch. A model can generate a non-standard token sequence that decodes to valid text but cannot survive a round-trip through the tokenizer. The root cause lies in the asymmetry between how a tokenizer encodes text and how a model generates tokens: encode (text to tokens) is one-to-one, while decode (tokens to text) is many-to-one. Multiple different token sequences can decode to the exact same string. So if you store the model's output as text and re-tokenize it for the next turn, you may get completely different token IDs.

2. Reasoning pruned by chat templates. Reasoning models like Qwen3 and Kimi K2 use chat templates with a "cut-thinking boundary" -- they drop historical assistant reasoning before the last user message. Agentic harnesses often inject User messages mid-task -- for example, the Terminus-2 harness uses User for terminal outputs, while other harnesses use it for engine retries like "Parse failed". Each injection pushes the cut-think boundary forward, silently erasing the reasoning that the model actually sampled.

3. Lossy chat-template re-rendering. When an inference engine re-applies the chat template on every call, small formatting differences accumulate. A tool call emitted as compact JSON ({"name":"bash",...}) gets parsed and re-serialized with spaces added after colons and commas. Same semantics, different bytes, different token IDs -- and a broken prefix.

How Miles enforces TITO end-to-end

Miles builds on slime but focuses on new hardware, large-scale MoE RL, and production-grade stability. Its TITO implementation is built from four interlocking components:

(1) Inference session server. Rather than reconstructing the token sequence from scratch each turn, the inference session server is a thin server layer that maintains per-trajectory state, keyed by session id. Under each id it holds a growing token buffer P that is appended in place every turn. The buffer also stores logprobs and routed expert indices, so it can be shipped directly to the trainer without any reconstruction.

(2) Append-only at three levels. Miles enforces the append-only invariant at the message list level, the chat-template rendering level, and the token sequence level. To prevent reasoning from being pruned, Miles ships fixed Jinja templates that disable cut-thinking via a clear_thinking: false kwarg, preserving historical reasoning across turns.

(3) Pluggable TITO tokenizer. Instead of retokenizing the whole sequence each turn, Miles tokenizes only the newly appended messages and splices the resulting IDs onto the buffer. But real models need small per-model patches at the splice boundary:

• Qwen3: The model stops generating at <|im_end|>, but the canonical template appends a trailing \n token. Miles inserts the missing newline before splicing.
• GLM-4.7: The model samples a role-boundary token as both a stop token and a next-message-start token. If the harness injects a different role next, Miles overwrites the wrong boundary token with the correct one -- with its loss mask zeroed so the swap is never trained on.

(4) TokenSeqComparator. After each rollout, Miles verifies the token buffer matches what a fresh render of the full message list would produce. It uses a hybrid text-token check: structural comparison at special-token boundaries, and string comparison between them, so harmless retokenization differences don't trigger false alarms. Two verification scripts run this comparator across all supported model and append-role-set combinations: a CPU/fast layer on rendered token sequences, and a GPU/e2e layer that repeats the check under real model inference. Either failing blocks the change before it reaches training.

Enabling TITO in your own pipeline

TITO is available now in the Miles framework, which is open-source. Turning it on requires three flags:

ROLLOUT_ARGS+=(
   --use-session-server
   --hf-checkpoint Qwen/Qwen3-4B
   --tito-model qwen3
   --tito-allowed-append-roles tool user
)

The --tito-model flag selects the model family so Miles can auto-resolve the correct fixed template. The --tito-allowed-append-roles flag declares which roles your harness will inject after the first assistant turn -- Miles rejects any undeclared role at runtime. The full TITO documentation covers verification scripts and how to onboard new models.

What models are supported

The TITO pipeline currently covers a broad set of frontier model families:

• Qwen: Qwen3, Qwen3.5, Qwen3-Next
• GLM: GLM-4.7, GLM-5, GLM-5.1
• Kimi: Kimi-K2, Kimi-K2.5, Kimi-K2.6
• Nemotron: Nemotron-3 Super and Ultra
• Minimax: Minimax-M2.5, Minimax-M2.7
• DeepSeek: DeepSeek-V3.2, DeepSeek-V4 (tool-only surface for now)

Adding a new model is deliberately cheap: it requires a fixed Jinja template plus a small merge_tokens override, verified by the two comparator scripts. GLM-5 adopted the TITO approach: the rollout engine emits the exact token IDs and metadata it produced, and the trainer consumes them directly. The approach is also gaining traction beyond Miles -- for agentic RL, the inference server should be a simple Token-In, Token-Out endpoint, with every other operation -- chat template application, parsing, reasoning extraction, tool-call handling, multi-turn stitching, and loss-mask construction -- happening in client code you control and can unit-test.

What this changes for the field

The assumption that "training and inference use the same model weights, so they're on-policy" turns out to be incomplete. Token identity -- not just weight identity -- is what determines whether a rollout can be faithfully reproduced and trained on. This is not simply design-pattern purism. In multi-turn RL, token identity determines whether a rollout can be reproduced, packed, and trained on efficiently.

The practical consequence is significant: any agentic RL pipeline that reconstructs the training sequence from stored text -- rather than preserving raw token IDs through the rollout -- is silently introducing off-policy bias. The longer the trajectories and the more tool calls involved, the worse the drift. TITO reframes this as an infrastructure problem with a concrete, verifiable solution, not just a theoretical concern to acknowledge and move on from.