Liquid AI's LFM2.5 Upgrades Its Own Tokenizer, Hitting 3.7x Faster On-Device Speed

Tokenizers get baked in at the start of training and rarely revisited. Liquid AI just challenged that assumption. They've published a recipe for upgrading a pretrained model's tokenizer in place, doubling LFM2.5-8B-A1B's vocabulary from 65K to 128K tokens without discarding the original training run. The result: dramatically faster on-device inference for languages that were previously getting a raw deal.

The tokenizer tax on non-English languages

A tokenizer is fixed before pre-training begins, and its vocabulary reflects whatever languages dominated the training corpus at that time. Languages that were underrepresented get split into far more tokens per word. Because a language model runs its decoder once per output token, more tokens per word means more steps, more latency, and more energy. For on-device models, the cost compounds further.

On a large cloud model, the embedding matrix and the output projection (the LM-head, which maps hidden states back to vocabulary probabilities) are a small fraction of total parameters. On a compact edge model, they dominate per-token memory bandwidth. At batch size 1, typical on-device, the LM-head reads the entire vocabulary on every generation step. A larger vocabulary means a larger matrix to stream on each token and keep in RAM, so edge models ship compact vocabularies and accept fragmentation for languages outside their priority set.

LFM2's original 65K byte-level BPE tokenizer was built for English, code, and a fixed set of languages, leaving little budget for Hindi, Vietnamese, or Thai. BPE (Byte Pair Encoding) is the standard algorithm for building tokenizer vocabularies: it starts from individual bytes and repeatedly merges the most frequent adjacent pairs into new tokens, building up a vocabulary of common subword units. Liquid AI wanted to fix the multilingual gap on a checkpoint they had already trained, without starting over.

Grow the tokenizer, don't replace it

If you own the tokenizer, you can grow it rather than swap it out. The new tokenizer is seeded with the original merge rules, which are frozen, then BPE training continues on a multilingual corpus. Two useful properties follow: most of the original 65K tokens carry over unchanged, and every new token has an exact decomposition into a sequence of original tokens.

That exact decomposition makes initialization clean. Tokens that carry over keep their original embedding row. Each new token gets the mean of its sub-tokens' rows. Nothing is initialized randomly, and there is no cross-tokenizer alignment to work out. New representations are composed from what the model already learned, not guessed from scratch.

Two-stage adaptation

Swapping the tokenizer cold degrades the model significantly. Training every parameter at once damaged the parts that already worked, so the team split adaptation into two stages, both run before the usual mid-training and post-training phases.

• Stage 1, Embedding only: Train only the new embedding rows on 600B tokens with the rest of the model frozen. The new tokens settle without disturbing the body, and this alone recovers most of the quality lost at the swap.
• Stage 2, Full continued pre-training: Unfreeze everything and continue pre-training on 400B tokens of a balanced multilingual mixture. This folds the new vocabulary into the body of the model and closes the remaining gap.

The cold swap costs 5.8 aggregate points. Stage 1 wins back 4.8 of them, and stage 2 recovers the rest and pushes past the source. The final model scores higher than the original checkpoint on the eight-benchmark aggregate, though the team attributes most of that surplus to the extra 400B tokens of pre-training rather than the vocabulary change itself.

Compression numbers by language

The payoff is substantial for languages that were previously over-fragmented:

• Thai: 4.0x fewer tokens
• Bengali: 3.4x fewer tokens
• Vietnamese: 2.6x fewer tokens
• Hindi: 2.4x fewer tokens
• English and code: essentially unchanged (1.01x)

Fewer tokens per word doesn't automatically mean faster output, because the bigger vocabulary makes each token step slightly slower. Going from 65K to 128K slows per-token decode by 7 to 10% on reference devices (an M4 Max CPU and GPU, and a Snapdragon 8 Elite Gen 5), since the larger LM-head is read on every step. But what a user perceives is time per character, not time per token. Once you account for compression gains, the net result is a 2.2 to 3.7x per-character decode speedup for under-served languages across reference devices.

The team stopped at 128K because pushing the vocabulary higher costs far more throughput on phones than the extra compression yields: up to 37% slower at 256K on the Snapdragon.

Benchmark gains, no regressions

On Global-MMLU (a multilingual knowledge benchmark), previously under-tokenized languages improve substantially while already-supported languages hold steady:

• Vietnamese: +11.6 points
• Indonesian: +9.1 points
• Hindi: +7.6 points
• Korean, Arabic, Chinese, Japanese: flat or marginal changes

The accuracy gains come from more than tokenization efficiency. Fewer tokens means the model's fixed context window holds more content, and the model does less work to process each word, which translates into better comprehension on longer inputs.

When this recipe applies

This approach fits a specific case: you own the tokenizer and can continue its original BPE merges, which requires access to the original merge rules and special-token configuration. Moving to an off-the-shelf third-party tokenizer calls for zero-shot transfer methods instead. The approach also requires continued pre-training, so it pays off most when the original training run was expensive enough that reusing it beats starting from scratch.

The broader field has generally treated tokenizers as immutable artifacts: pick one before training and live with it forever. When deployment priorities shift, languages added later are split into many more tokens per word, raising latency, compute, and energy consumption for users of those languages. This work shows the tokenizer doesn't have to be a permanent constraint, at least when you control its design.

Getting started

LFM2.5-8B-A1B is released with open weights on Hugging Face under the LFM Open License v1.0, with deployment guides for llama.cpp, MLX, vLLM, and SGLang. The expanded 128K tokenizer ships alongside the weights at no cost. The full method, ablations, and two approaches that looked reasonable but broke the model are documented in the technical report on arXiv.

You can load it directly with Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "LiquidAI/LFM2.5-8B-A1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="bfloat16"
)
input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is C. elegans?"}],
    add_generation_prompt=True,
    return_tensors="pt",
    tokenize=True,
).to(model.device)
output = model.generate(input_ids, max_new_tokens=512)

The official blog post walks through the full recipe with figures for each stage. If you're building multilingual applications targeting on-device deployment in Southeast Asia or South Asia, the technique is also a template for anyone locked into a tokenizer that no longer reflects their actual user base.