Liquid AI's toktoktok Ships Production Code Without Engineers Reading a Line
Liquid AI open-sourced toktoktok, a production-grade BPE tokenizer trainer written entirely by coding agents inside an iteration loop against real data.
- Liquid AI open-sourced toktoktok, a BPE tokenizer trainer written entirely by coding agents
- Both Claude Opus 4.5 and Codex GPT-5.2 zero-shot a toy trainer in 30 minutes, neither scaled
- An iteration loop against real production data and external verification carried Claude Opus 4.5 to production
- Output is tiktoken-compatible, handles trillions of tokens on one machine within a declared memory budget
- Supports multi-phase training, warm start from existing tokenizers, and
.txt/.parquetcorpora - Full writeup argues loop design now matters more than raw model capability
Liquid AI ran an experiment with a deceptively simple question: can coding agents autonomously ship production-grade software without a human ever reading the code? The answer, published alongside an open-source tokenizer trainer called toktoktok, is yes, but only when the agent runs inside a well-designed loop against real production data.
The team needed a byte-pair encoding (BPE) tokenizer trainer that could chew through trillions of tokens on a single machine for their vocabulary-size research on edge LLMs. Existing options fell short: sentencepiece is slow for BPE, Hugging Face tokenizers ran out of memory on their corpora, and tiktoken cannot train at all. So instead of staffing engineers, they handed the spec to two agents (Claude Opus 4.5 and Codex with GPT-5.2) and let them work.
Zero-shot got them a toy, not a product
Both agents produced a working trainer inside 30 minutes. Configs parsed, corpora walked, merges applied, and a valid .tiktoken file dropped out the other end. All unit tests passed. On a few megabytes of clean text, the runs looked like unambiguous wins.
Then the trainers met the real corpus and broke in ways that toy data cannot surface:
- Parquet files with mixed encodings were silently mishandled
- Per-document Vec overhead blew out memory at roughly 1 percent of the target corpus