Ai2's OLMo Hybrid Beats Transformers on Meaning While Using 49% Fewer Tokens

The transformer has ruled language modeling for years, but a new class of architectures is mounting a serious challenge: hybrid models that mix traditional attention layers with linear recurrent layers (modern RNNs). The question is no longer whether hybrids can match transformers on leaderboards , they can , but why, and on which specific tasks. Ai2 just published a technical report that answers this at the finest possible granularity: the individual token level.

The field is already moving

Hybrid language models , architectures that mix transformer attention with linear recurrent layers , have been gaining momentum across the field, with recent efforts from projects like Samba, Nemotron-H, Qwen3-Next, Kimi Linear, and Qwen 3.5. These models have been trained at scales up to 9B active parameters and 36T tokens with encouraging results. Yet despite the momentum, a fundamental question has gone unanswered: what, exactly, does each architectural component contribute to the model's predictions?

Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks and hybrid models that mix recurrence and attention , yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. Ai2's new study is a direct attempt to resolve that.

A controlled experiment, token by token

The key to this study's credibility is the experimental setup. Ai2 compared OLMo 3 (a pure transformer) and OLMo Hybrid (which swaps 75% of attention layers for Gated DeltaNet recurrent layers) in a head-to-head evaluation. The hybrid uses a 3:1 hybridization ratio, replacing the sliding-window attention layers from OLMo 3 with Gated DeltaNet layers. Because both models were built to be as alike as possible outside their architectures , matched on data, tokenizer, and training recipe , any difference in their predictions mostly reflects the architecture itself.

The team fed both models the same passages spanning articles, Wikipedia entries, books, scientific papers, Python, HTML, and LaTeX, then recorded each model's probability for the actual next token. The metric they track is the loss gap: the difference in prediction error between the two models. A positive gap means the hybrid predicted better; negative means the transformer did.

Where hybrids win: meaning-bearing words

The hybrid's advantage is real, but it concentrates on specific token types. The clearest divide is between content words (meaning-bearing nouns, verbs, adjectives) and function words ("the," "of," "is"). The hybrid predicts content words better than the transformer with a loss gap around 0.04, whereas the gap is closer to 0.02 on function words.

Why does this happen? The intuition is that content words require understanding the evolving context of a sentence , what topic is being discussed, which entity is the subject, what has changed since the last clause. Recurrent layers' efficiency at tracking evolving state gives hybrids an edge at long context lengths. Function words, by contrast, are largely predictable from local syntax, so any model can guess them with high confidence regardless of architecture.

The hybrid also has a pronounced edge on tokens that require coreference tracking , knowing which earlier noun a pronoun refers to. This is a classic state-tracking problem, exactly the kind of sequential reasoning that recurrent layers are theoretically suited for.

Where transformers hold their ground

The study surfaces two specific contexts where the hybrid's advantage disappears:

• Verbatim repetition: When the next token simply repeats a word or phrase already in the passage (an n-gram copy , a run of text that appeared verbatim earlier), the transformer matches or beats the hybrid. The longer the repeated run, the smaller the hybrid's lead, approaching zero for long repeated sequences. Attention is essentially doing a lookup , it can reach back to any earlier token exactly , and that's all you need when the answer is already sitting in the context.
• Closing brackets: Transformers match hybrids on closing brackets like ], ), and } across code, language, and markup , but not on opening brackets. The researchers note that attention is known to suffice for bracket matching, so recurrence adds nothing there.

The real breakthrough: filtered loss as an evaluation tool

Beyond the architectural insights, the study introduces something practically useful: token-type-filtered loss as an evaluation signal during pretraining. The idea is simple but powerful , instead of reporting a single average loss across all tokens, you report separate losses for different token categories. This lets you see architectural differences that a global average would hide.

To demonstrate this, the team ran the analysis on three 1B-parameter models trained from scratch: a pure transformer, a hybrid (attention + GDN), and a pure recurrent model with no attention at all. The results were striking:

• On meaning-bearing tokens that aren't repeats: hybrid and pure recurrent models overtake the transformer, with the hybrid performing best.
• On repeated tokens: the pure recurrent model , with no attention to reach back for the copy , falls behind both the hybrid and the transformer.
• These filtered token losses reveal different fine-grained differences between architectures early in training in a way that would not otherwise be visible.

This is a meaningful contribution for anyone running pretraining ablations. Global perplexity can make a hybrid and a transformer look nearly identical even when they have very different strengths. Filtered losses expose those differences early , potentially saving significant compute on architecture search.

What this means for architecture design

Theoretically, hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Overall, the results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.

The practical takeaway for teams building or evaluating LLM architectures:

• A single overall loss is too blunt to compare transformer and hybrid architectures meaningfully.
• If your use case involves tasks requiring semantic understanding, coreference, or state tracking (e.g., long-form generation, dialogue, document summarization), hybrids have a measurable edge.
• If your use case is heavy on verbatim retrieval from context , like RAG-style tasks where the answer is literally in the prompt , transformers hold their own.
• OLMo Hybrid matches OLMo 3's accuracy on MMLU while using 49% fewer training tokens, suggesting the efficiency gains from hybrid architectures are real and not just a benchmark artifact.

The full technical report and all model artifacts from OLMo Hybrid are open and available to explore. Ai2 says the next step is applying these token-level findings directly to pretraining evaluation pipelines , which could make filtered loss a standard tool in the architecture researcher's toolkit.