Nous Research Finally Explains Why Byte-Level Models Keep Losing to Tokenizers
Théo Gigant, Bowen Peng, and Jeffrey Quesnelle run seven controlled experiments to pinpoint exactly why subword models beat byte-level ones — and find two culprits that matter most

- Core finding: Of seven hypotheses for why subword LLMs beat byte-level ones, only two matter significantly at 1.7B scale: sample throughput and subword boundary priors.
- Throughput dominates: BPE compresses ~4.75 bytes/token, giving subword models ~4x more training samples per gradient step at equal FLOPs — simulating this alone yields a persistent loss improvement.
- Boundary signal is second: Start-of-subword markers act as a morphological inductive bias that persists after removal; end-of-subword markers help but create dependency.
- Four hypotheses fail: Vocabulary capacity, subword positional distances, per-subword loss weighting, and next-subword prediction show negligible or negative effects at this scale.
- Practical upshot: Byte-level models should prioritize throughput (patching) first, then boundary signal — the gap can be closed without a static tokenizer.
- Paper: arxiv.org/abs/2604.27263 by Théo Gigant, Bowen Peng, and Jeffrey Quesnelle (Nous Research); no code release.
Byte-level language models have a compelling pitch: no tokenizer, no vocabulary, no language-specific preprocessing. But they keep losing to subword models in practice, and for years the field has treated the gap as a known fact without a precise explanation. Subword tokenization is an essential part of modern large language models, yet its specific contributions to training efficiency and model performance remain poorly understood. A new paper from Théo Gigant, Bowen Peng, and Jeffrey Quesnelle , all at Nous Research , finally puts that intuition under a microscope.
The authors decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline, formulating and testing hypotheses across dimensions including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. The result is the clearest causal account yet of what subword tokenization is actually doing for you.
Seven suspects, one lineup
The paper, titled "Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation", frames the problem as a controlled experiment. Rather than comparing a byte-level model to a subword model and calling it a day, the authors inject each suspected advantage of subword tokenization , one at a time , into a byte-level pipeline and measure the change in validation loss.
The seven hypotheses they test fall into three buckets:
- Computational efficiency: Larger vocabulary parameters give the model more capacity (H1); subword compression means the model sees ~4x more raw text per gradient step at equal FLOPs (H2)
- Structural priors: Subword end-boundaries leak future bytes, making prediction easier (H3); subword start-boundaries act as a morphological inductive bias (H4); subword positional distances align attention with semantically meaningful units (H5)
- Optimization objective: Minimizing cross-entropy per subword is a better proxy than per-byte (H6); predicting the next subword is a form of multi-token prediction that improves representations (H7)
All three authors are affiliated with Nous Research. The main experiments use a 1.7B-parameter LLaMA-3 architecture trained on fineweb-edu, with 68M-parameter replications in the appendix for scale comparison. Each intervention is designed to introduce negligible compute overhead and leave the rest of the training loop untouched.
The two things that actually move the needle
Out of seven hypotheses, only three produce a meaningful improvement. And of those three, two dominate.
Sample throughput is the biggest lever. This is usually the biggest drawback that hinders the competitiveness of byte-level language models, such that state-of-the-art byte-level language models come with methods to compress the byte sequences and thus increase sample throughput closer to the subword counterpart. The LLaMA-3 BPE tokenizer compresses fineweb-edu at roughly 4.75 bytes per token, meaning a subword model sees about 4x more training samples per gradient step at equal FLOPs. To simulate this in a byte model, the authors sum the embeddings of every 4 contiguous bytes into a single latent token for the first 50k of 100k training steps, predicting only the first byte of the next chunk, then revert to standard byte-level training. The throughput-warmed model immediately drops below the baseline curve when the intervention is removed and stabilizes at a persistent lower loss for the remaining 50k steps.
Subword boundaries are the second meaningful effect. Adding end-of-subword markers as input embeddings produces a large gain throughout training , because end-boundaries effectively leak future bytes (whitespace always follows an end-boundary, for example). Start-of-subword boundaries also help, but for a different reason: they don't leak future information, so when the boundary signal is removed after 50k steps, the improvement persists. The authors interpret this as start-boundaries supplying a morphological inductive bias , a structural hint about how words are built , while end-boundaries supply a near-term prior the model becomes dependent on and can't generalize without.
Here is a sketch of how the throughput intervention works in practice:
# Throughput warm-up: compress 4 bytes into 1 latent token
# Input shape: (batch, 4*L) -> reshape to (batch, L, 4)
chunks = byte_sequence.reshape(batch_size, seq_len, 4)
# Sum embeddings of the 4 bytes in each chunk
latent_tokens = embed(chunks).sum(dim=-1) # (batch, L, d_model)
# Predict only the first byte of the next chunk
loss = cross_entropy(model(latent_tokens), next_chunk_first_byte)
# After 50k steps: revert to standard byte-level training
# The model retains a persistent loss advantageWhat doesn't matter (at this scale)
Four of the seven hypotheses either hurt or show negligible effect at 1.7B parameters:
- Vocabulary capacity (H1): Adding 70M parameters of n-gram embedding tables to match a subword model's vocabulary size produces only a small improvement , not nearly enough to explain the gap.
- Subword distances in positional encoding (H5): Replacing byte-position RoPE with subword-position RoPE helps slightly as a prior but doesn't survive as an inductive bias.
- Cross-entropy per subword (H6): Reweighting the loss to optimize per-subword rather than per-byte shows minimal effect at this scale.
- Next-subword prediction (H7): Training the byte model to predict the next subword (analogous to multi-token prediction) actually performs worse than standard next-byte prediction at 1.7B.
The authors are careful to note that some of these null results could flip at larger scales. The vocabulary capacity effect, for instance, showed a larger gap at 68M parameters.
What this actually unlocks
The decomposition matters more than any single result. Before this paper, the byte-level vs. subword debate was largely a black box , you knew subword was better, but you couldn't say which property was doing the work. Now you can.
By simulating these effects in a byte-level setting, the authors refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Concretely, the findings suggest:
- For byte-level pretraining: prioritize throughput first (patch-based compression, as in Byte Latent Transformer), then add boundary signal second , either as an explicit prior or a training-time inductive bias, without needing a static tokenizer.
- For subword pretraining: the vocabulary-capacity argument is weaker than commonly framed at 1.7B. The compression-driven throughput multiplier is the load-bearing source of gain.
- The boundary benefit can be recovered without a static tokenizer by treating boundaries as a target (as in Bolmo) or a prior injected only during early training.
Subwords created with Unigram, and BPE to a lesser extent, align well with morphological reference segmentations, explaining why they provide a useful inductive bias during byte-level language model pretraining. This also means the boundary benefit may be weaker for languages with different morphological structures , an important caveat for multilingual work.
Limitations worth knowing
The study has real constraints. Each intervention was applied for only the first 50k of 100k training steps, so it's an open question whether the gains compound, plateau, or diminish when maintained throughout a full pretraining run. The experiments also test each effect in isolation , the paper explicitly does not account for the interaction between throughput gains and boundary priors, which likely compound in real subword training. And everything is English-only on fineweb-edu, so multilingual conclusions require more work.
No code or model weights are released alongside the paper, but the paper is available on Hugging Face and the full methodology is described in enough detail to replicate. If you're building a byte-level model or trying to understand why your tokenizer-free experiments keep underperforming, this is the most actionable diagnostic framework available.