Yale and Google DeepMind's Insertion Process Teaches AI to Write Without Left-to-Right Order
The Insertion Process learns where, what, and when to insert tokens, using variational inference over permutations to discover data-driven generation orders without left-to-right constraints.
Takeaways
- New framework: The Insertion Process (IP) learns to generate variable-length sequences by inserting tokens at arbitrary positions in a data-driven order.
- Core innovation: A bijective mapping between insertion trajectories and permutations enables exact likelihood computation, trained via permutation-based variational inference (ELBO).
- Three joint decisions: At each step the model learns where to insert, what token to place, and when to stop, all simultaneously.
- Tested on: Goal-conditioned maze planning and molecular SMILES string generation, both domains without a natural left-to-right structure.
- Key result: Learning insertion order improves both modeling quality and generalization over fixed or random order baselines in both domains.
- No code released: No public implementation or pretrained model is available alongside the paper at this time.
Every large language model you have ever used generates text the same way: left to right, one token at a time, committed to each choice before seeing what comes next. That constraint is so deeply baked into modern AI that it is easy to forget it is a choice, not a law of nature. A new paper from researchers at Yale, Google DeepMind, and the Gatsby Computational Neuroscience Unit proposes a principled alternative: the Insertion Process (IP), a generative model that learns to build sequences by inserting tokens at arbitrary positions, in an order it discovers for itself.
The tyranny of left-to-right
Autoregressive models (ARMs) are powerful, but their sequential commitment creates real problems for tasks where the natural structure is not linear. Autoregressive language models dominate modern NLP due to their strong likelihood-based training and reliable left-to-right decoding, but the strictly sequential nature of AR decoding entails two fundamental limitations: high inference latency and constrained throughput, and for tasks requiring global constraints or non-monotonic planning, a fixed chronological order may not be the most natural solution path.
The alternative that has attracted the most attention is masked diffusion. Masked Diffusion Models (MDMs) address some of these limitations, but the process of unmasking multiple tokens simultaneously in MDMs can introduce incoherences, and MDMs cannot handle arbitrary infilling constraints when the number of tokens to be filled in is not known in advance. In other words, MDMs require you to know the final sequence length upfront, a rigid assumption that breaks down in drug discovery, planning, and code generation.
Insertion-based models offer a third path. Insertion Language Models learn to insert tokens at arbitrary positions in a sequence, addressing limitations of both autoregressive models and Masked Diffusion Models. But there is a catch: the action space of insertion models is much larger than that of autoregressive models, making the learning challenging. And crucially, prior insertion models either hard-coded a fixed generation order or operated on a fixed-length canvas, never letting the model figure out the best order on its own.