LiquidAI's LFM2.5-Encoder Beats ModernBERT at Long Context 3.7x Faster on CPU

Encoders power the bulk of production NLP. Every intent classifier, safety filter, and document router running all day on your infrastructure is almost certainly an encoder. LFM2.5-Encoder-230M and LFM2.5-Encoder-350M are Liquid AI's first general-purpose encoders, built around one specific claim: fast at long context, even on CPU.

The quadratic wall

Standard transformer encoders like BERT and ModernBERT have attention costs that scale quadratically with input length. That's fine at 512 tokens. Push to 8,192 tokens on a CPU and they stall. At that length, ModernBERT-base takes over 90 seconds per forward pass. LFM2.5-Encoder-230M takes about 28 seconds. That 3.7x gap is the practical reason to pay attention to this release.

Built from a decoder, flipped for understanding

Liquid AI initializes each encoder from its corresponding LFM2 decoder backbone, then converts the causal decoder into a bidirectional encoder with three targeted changes:

• Bidirectional attention mask: each token attends to tokens on both sides, not just the ones before it.
• Non-causal short convolutions: convolutions are padded symmetrically so each token mixes in neighbors on both sides.
• Masked language modeling: 30% of tokens are masked during training, with a two-stage schedule that extends the context window to 8,192 tokens.

The LFM2 backbone interleaves gated short-convolution blocks with grouped-query attention. Convolutions scale linearly with sequence length, which is what keeps cost manageable at 8k tokens.

Benchmark results

LFM2.5-Encoder-350M ranks fourth among 14 models across 17 tasks drawn from GLUE, SuperGLUE, and multilingual classification. The three models ahead of it are all larger, including a 3.5B model nearly 10 times its size. The 230M variant beats ModernBERT-base and every EuroBERT model while being smaller than most of them.

On GPU, the picture is similar but narrower. ModernBERT-base leads below roughly 1K tokens on Apple GPU, but the LFM2.5-Encoders pull ahead from about 2K tokens onward. The speed advantage is sharpest on CPU, though it appears on GPU too for longer sequences.

Where these models fit

LFM2.5-Encoders are well-suited to high-volume understanding tasks, such as classification, routing, extraction, or scoring, that run continuously and need to stay cheap. A fine-tuned encoder is smaller, faster, and far cheaper than a generative LLM for these jobs, and it runs on CPUs you already own.

Liquid AI shipped five live demos, all running on CPU-only Hugging Face Spaces:

• Zero-shot prompt routing: define routing lanes as free text; the model scores a full prompt against every lane in one pass.
• Zero-shot policy linting: check text against company rules written in plain language, scored token by token.
• Spell checking, PII detection across 16 languages, and a masked-diffusion chatbot that generates text by iteratively unmasking rather than left-to-right.

Getting started

Both models are on Hugging Face under the LFM Open License v1.0. Loading follows standard Hugging Face conventions, but trust_remote_code=True is required because the bidirectional LFM2 architecture is not yet in the core transformers library.

from transformers import AutoModel
# Load encoder body, attach your own classification head
body = AutoModel.from_pretrained(
    "LiquidAI/LFM2.5-Encoder-230M",  # or 350M for higher accuracy
    trust_remote_code=True
)
# Fine-tune end-to-end with lr=1e-5 to 5e-5, bf16, 3-20 epochs

GPU users can install flash-attn alongside for better throughput. Choose LFM2.5-Encoder-350M when accuracy is the priority; choose LFM2.5-Encoder-230M for tighter hardware or higher throughput. Both support 15 languages and an 8,192-token context window out of the box.

The practical upshot: long-document tasks like contract analysis, support thread triage, or policy compliance checks can run on a laptop CPU in under 30 seconds per document. For teams currently routing those workloads through a GPU cluster or a hosted LLM API, that changes the infrastructure math considerably.