Tokenizers are one of those things that get baked in at the start of training and rarely revisited. Liquid AI just challenged that assumption. They've published a recipe for upgrading a pretrained model's tokenizer in place, doubling LFM2.5-8B-A1B's vocabulary from 65K to 128K tokens without throwing away the original training run. The result: dramatically faster on-device inference for languages that were previously getting a raw deal.
The tokenizer tax on non-English languages
A tokenizer is fixed before pre-training begins, and its vocabulary reflects whatever languages dominated the training corpus at that time. Languages that were underrepresented end up being split into far more tokens per word. This isn't just an aesthetic problem. Because a language model runs its decoder once per output token, more tokens per word means more steps, more latency, more energy. For on-device models, it's even worse.
On a large cloud model, the embedding matrix and the output projection (the LM-head, which maps hidden states back to vocabulary probabilities) are a small fraction of total parameters. On a compact edge model, they dominate per-token memory bandwidth. At batch size 1, which is typical on-device, the LM-head reads the entire vocabulary on every single generation step. A larger vocabulary means a larger matrix to stream on each token and to keep in RAM, so edge models ship compact vocabularies and live with fragmentation outside their priority languages.
LFM2's original 65K byte-level BPE tokenizer was built for English, code, and a fixed set of languages, leaving little budget for Hindi, Vietnamese, or Thai. BPE (Byte Pair Encoding) is the standard algorithm for building tokenizer vocabularies: it starts from individual bytes and repeatedly merges the most frequent adjacent pairs into new tokens, building up a vocabulary of common subword units. The team wanted to fix the multilingual gap on a checkpoint they had already trained, without starting over.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves