Edinburgh's MIXAR Ditches Tokenizers to Read 8 Languages as Images
A new autoregressive model reads and writes text as pixels across eight languages, dodging tokenizers entirely while surviving visual attacks on input.
ProTakeaways
- MIXAR is the first generative pixel-based LLM trained across 8 languages spanning Latin and CJK scripts (paper).
- Scales to 477M parameters and uses 32x32 patches to handle Chinese, Japanese, and Korean characters.
- Two-stage training combines MLE reconstruction with a GAN objective to sharpen generated patches.
- Beats MGPT-2 and PIXEL-M4 on XNLI multilingual accuracy, hitting 67.4 non-English average.
- Far more robust to orthographic attacks than GPT-2, degrading gracefully as attack ratio increases.
- Still weak on CJK generation and single-font brittleness; code, weights, and data pipeline to be released.
Every mainstream large language model starts by chopping text into tokens, and every one of them pays a tax for that choice. Vocabularies balloon when you add languages, low-resource scripts get shortchanged, and adversaries can slip past filters by swapping a Latin letter for a look-alike Cyrillic one. A team from the University of Edinburgh has pushed the pixel-based alternative one step further with MIXAR, the first generative pixel-based language model trained on eight languages spanning several scripts.
The core idea is to skip tokenization altogether. Render the text as an image, then have a transformer decoder predict the next patch of pixels autoregressively. That single move makes the model script-agnostic, keeps the parameter budget from exploding as you add languages, and gives you built-in resistance to character-level visual attacks.
The tokenizer tax
Token embeddings scale linearly with vocabulary size. A monolingual BERT sits around 23M parameters just for its embedding matrix; the multilingual variant balloons to roughly 81M, and Qwen3-8B pushes past 600M. That cost is only the beginning. Fixed vocabularies systematically penalize low-resource languages with higher per-word token counts and degraded quality.
Token-based models also inherit a set of well-known pathologies, from under-trained tokens that behave unpredictably, to unargmaxable outputs, to susceptibility to prompt-injection style vocabulary attacks. Pixel-based predecessors like PIXEL and PIXAR attacked those problems, but only in English. A more recent effort, PIXEL-M4, added multilingual coverage but was BERT-like and encoder-only, so it could classify text without generating any.
How MIXAR actually works
MIXAR is a decoder-only transformer that consumes and produces text as a stream of rendered image patches. Text is drawn with the PangoCairo tool using the pixel-style Pixeloid Sans font, then sliced into square patches that feed the model. The final layer projects hidden states back into pixel space, and a sigmoid squashes them into binary black-and-white patches.