---
title: "Yale and Google DeepMind's Insertion Process Teaches AI to Write Without Left-to-Right Order"
subtitle: "The Insertion Process learns where, what, and when to insert tokens, using variational inference over permutations to discover data-driven generation orders without left-to-right constraints."
url: "https://alphasignal.ai/news/yale-and-google-deepmind-s-insertion-process-teaches-ai-to-write-without-left"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-01T00:00:00.000Z"
updated: "2026-06-09T13:00:18.327Z"
type: "Paper"
topics:
  - "Llms"
  - "Post Training"
subtopics:
  - "Structured Output"
  - "Distillation"
upvotes: 1945
source_url: "https://arxiv.org/abs/2606.02133"
comments: 0
truncated: true
---

# Yale and Google DeepMind's Insertion Process Teaches AI to Write Without Left-to-Right Order

> The Insertion Process learns where, what, and when to insert tokens, using variational inference over permutations to discover data-driven generation orders without left-to-right constraints.

Every large language model you have ever used generates text the same way: left to right, one token at a time, committed to each choice before seeing what comes next. That constraint is so deeply baked into modern AI that it is easy to forget it is a choice, not a law of nature. A new paper from researchers at Yale, Google DeepMind, and the Gatsby Computational Neuroscience Unit proposes a principled alternative: the [Insertion Process (IP)](https://arxiv.org/abs/2606.02133), a generative model that learns to build sequences by inserting tokens at arbitrary positions, in an order it discovers for itself.

## The tyranny of left-to-right

Autoregressive models (ARMs) are powerful, but their sequential commitment creates real problems for tasks where the natural structure is not linear. Autoregressive language models dominate modern NLP due to their strong likelihood-based training and reliable left-to-right decoding, but the strictly sequential nature of AR decoding entails two fundamental limitations: high inference latency and constrained throughput, and for tasks requiring global constraints or non-monotonic planning, a fixed chronological order may not be the most natural solution path.

The alternative that has attracted the most attention is masked diffusion. Masked Diffusion Models (MDMs) address some of these limitations, but the process of unmasking multiple tokens simultaneously in MDMs can introduce incoherences, and MDMs cannot handle arbitrary infilling constraints when the number of tokens to be filled in is not known in advance. In other words, MDMs require you to know the final sequence length upfront, a rigid assumption that breaks down in drug discovery, planning, and code generation.

Insertion-based models offer a third path. Insertion Language Models learn to insert tokens at arbitrary positions in a sequence, addressing limitations of both autoregressive models and Masked Diffusion Models. But there is a catch: the action space of insertion models is much larger than that of autoregressive models, making the learning challenging. And crucially, prior insertion models either hard-coded a fixed generation order or operated on a fixed-length canvas, never letting the model figure out the best order on its own.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Variational Learning for Insertion-based Generation (arXiv 2606.02133)](https://arxiv.org/abs/2606.02133)
- [Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions (arXiv 2505.05755)](https://arxiv.org/abs/2505.05755)
- [Insertion Based Sequence Generation with Learnable Order Dynamics (arXiv 2602.18695)](https://arxiv.org/abs/2602.18695)
- [Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes (arXiv 2603.23507)](https://arxiv.org/pdf/2603.23507)
- [Parallelism and Generation Order in Masked Diffusion Language Models (arXiv 2601.15593)](https://arxiv.org/html/2601.15593)