Liquid AI's toktoktok Ships Production Code Without Engineers Reading a Line

Liquid AI ran an experiment with a deceptively simple question: can coding agents autonomously ship production-grade software without a human ever reading the code? The answer, published alongside an open-source tokenizer trainer called toktoktok, is yes, but only when the agent runs inside a well-designed loop against real production data.

The team needed a byte-pair encoding (BPE) tokenizer trainer that could chew through trillions of tokens on a single machine for their vocabulary-size research on edge LLMs. Existing options fell short: sentencepiece is slow for BPE, Hugging Face tokenizers ran out of memory on their corpora, and tiktoken cannot train at all. So instead of staffing engineers, they handed the spec to two agents (Claude Opus 4.5 and Codex with GPT-5.2) and let them work.

Zero-shot got them a toy, not a product

Both agents produced a working trainer inside 30 minutes. Configs parsed, corpora walked, merges applied, and a valid .tiktoken file dropped out the other end. All unit tests passed. On a few megabytes of clean text, the runs looked like unambiguous wins.

Then the trainers met the real corpus and broke in ways that toy data cannot surface:

• Parquet files with mixed encodings were silently mishandled
• Per-document Vec overhead blew out memory at roughly 1 percent of the target corpus
• Only part of the critical path was parallelized, so cores stayed busy while throughput stayed unacceptable
• A \s+(?!\S) pre-tokenization regex triggered quadratic backtracking on adversarial whitespace
• Non-contiguous rank ordering made vocabularies encode differently than intended
• Rust's regex crate reparsed {1,3}+ as ({1,3})+, breaking number encoding in a way only visible when checked against tiktoken

The fix was not a smarter prompt. It was a loop: execute against the real dataset, hit a wall, let the agent diagnose the symptom, patch, run again. After more than five iterations with little progress, the Codex/GPT-5.2 track was shelved; Claude Opus 4.5 closed out the remaining issues over a handful more turns and produced a trainer that ran the full multilingual-plus-code configuration in a few days on an AMD EPYC 9755 with 256 threads and 2 TB of RAM.

The two design rules that made autonomy safe

The writeup makes an argument that matters more than the artifact: the operator never read a line of code. That was only tolerable because two things were true.

1. The spec targeted a multi-domain expert. The AGENTS.md file stated outcomes and constraints, not mechanisms. The agent already knew both OpenAI's cl100k regex and Rust's rayon, a combination Liquid's own engineers do not have in one head. Writing longer specs for humans risks becoming pseudocode; for LLMs, short intent-level specs work because the background is preinstalled.
2. Verification lived outside the agent's reach. Correctness was defined by whether the trained vocabulary loaded in tiktoken and Hugging Face tokenizers and produced identical token IDs across languages, numbers, currency formatting, tabs, CRLF endings, and source code. The agent could not edit those libraries or its way past them.

What toktoktok actually gives you

The repo ships under Apache 2.0 and produces tiktoken-compatible vocabularies that convert losslessly to Hugging Face tokenizers. Notable features for anyone training their own model:

• Memory-bounded reservoir sampling. Declare a working_set_mb budget and corpora larger than RAM stay fairly represented. A 4 GB working set handles roughly 20 GB of source text; 64 GB scales past 100 GB.
• Multi-phase training. Allocate explicit merge budgets per language or domain so the largest corpus does not eat the whole vocabulary.
• Warm start. Extend an existing tokenizer with new merges without invalidating existing token IDs, which means you can grow vocabulary for a new domain without retraining your embedding table.
• 1,161 hardcoded merges reserve ranks for all two- and three-digit numbers, whitespace runs, , common programming operators, and ellipses, so numeric and whitespace encoding stays consistent regardless of corpus statistics.
• Input flexibility. Reads .txt and .parquet directories recursively; parquet files need a text column.

Training is driven by a YAML config passed with -c. A minimal run looks like:

cargo build --release
./target/release/toktoktok -c examples/from_scratch.yaml
python scripts/test_tokenizer.py tokenizer.tiktoken examples/special_tokens.txt

Why this is worth reading even if you never train a tokenizer

The real claim is not about tokenizers. It is that the bottleneck for autonomous coding agents has shifted from model capability to loop design. The two thousand lines in the repo are not the residue of a clever zero-shot; they are the residue of an iterative process where each non-obvious line was cheap to write once the loop had discovered it needed writing. Liquid AI now runs similar loops for kernel tuning, CI monitoring, pull request triage, and production log anomaly scanning: checking the metric instead of the code. If your team is still evaluating agents on prototype tasks with self-reported success, the interesting benchmark has moved.