Liquid AI's LFM2.5 Upgrades Its Own Tokenizer, Hitting 3.7x Faster On-Device Speed
Liquid AI doubled LFM2.5-8B-A1B's tokenizer to 128K without retraining, cutting Thai tokens by 4x and boosting on-device decoding speed by up to 3.7x.
- Liquid AI doubled LFM2.5-8B-A1B's tokenizer from 65K to 128K vocabulary without retraining from scratch.
- Thai gets 4.0x fewer tokens, Bengali 3.4x, Vietnamese 2.6x, Hindi 2.4x; English and code stay unchanged.
- On-device per-character decoding is estimated 2.2 to 3.7x faster for under-served languages on M4 Max and Snapdragon 8 Elite.
- Two-stage adaptation: freeze the model and train only new embedding rows (600B tokens), then unfreeze for full continued pre-training (400B tokens).
- Global-MMLU gains: Vietnamese +11.6, Indonesian +9.1, Hindi +7.6; previously supported languages hold steady.
- Model weights, 128K tokenizer, and technical report are all publicly available now.
Tokenizers get baked in at the start of training and rarely revisited. Liquid AI just challenged that assumption. They've published a recipe for upgrading a pretrained model's tokenizer in place, doubling LFM2.5-8B-A1B's vocabulary from 65K to 128K tokens without discarding the original training run. The result: dramatically faster on-device inference for languages that were previously getting a raw deal.
The tokenizer tax on non-English languages
A tokenizer is fixed before pre-training begins, and its vocabulary reflects whatever languages dominated the training corpus at that time. Languages that were underrepresented get split into far more tokens per word. Because a language model runs its decoder once per output token, more tokens per word means more steps, more latency, and more energy. For on-device models, the cost compounds further.
On a large cloud model, the embedding matrix and the output projection (the LM-head, which maps hidden states back to vocabulary probabilities) are a small fraction of total parameters. On a compact edge model, they dominate per-token memory bandwidth. At batch size 1, typical on-device, the LM-head reads the entire vocabulary on every generation step. A larger vocabulary means a larger matrix to stream on each token and keep in RAM, so edge models ship compact vocabularies and accept fragmentation for languages outside their priority set.
LFM2's original 65K byte-level BPE tokenizer was built for English, code, and a fixed set of languages, leaving little budget for Hindi, Vietnamese, or Thai. BPE (Byte Pair Encoding) is the standard algorithm for building tokenizer vocabularies: it starts from individual bytes and repeatedly merges the most frequent adjacent pairs into new tokens, building up a vocabulary of common subword units. Liquid AI wanted to fix the multilingual gap on a checkpoint they had already trained, without starting over.