Northwestern Catches LLMs Recycling the Same Code in Evolutionary AI
A new study finds LLMs used as mutation operators in genetic programming reliably converge to the same structural templates, undermining the diversity that evolutionary search depends on.
- Core finding: LLMs used as mutation operators in evolutionary algorithms reliably converge to a small set of structural templates, regardless of model or prompt.
- The numbers: In 87% of mutation chains, over 93% of mutations revisit a previously seen structural skeleton -- only constants change.
- Control comparison: Classical GP subtree mutation does NOT show comparable convergence, confirming the effect is intrinsic to LLMs.
- Reasoning helps: Reasoning-enabled model variants (GPT-5 Mini with reasoning, Gemini with thinking) produce significantly more structural diversity than base models.
- Practical impact: Systems like FunSearch, AlphaEvolve, and QD algorithms using LLM mutation may be exploring far less of program space than assumed.
- Code available: Full experimental code is open-source at github.com/can-gurkan/lmca.
There is a growing wave of systems that use LLMs as mutation engines inside evolutionary algorithms -- from Evolution through Large Models to Google's AlphaEvolve and FunSearch. The pitch is compelling: instead of random bit-flips, you get semantically intelligent edits that understand what a program does. But a new paper from Northwestern University and Augustana College asks a deceptively simple question: when you keep asking an LLM to mutate a program, does it actually explore new territory, or does it just keep rewriting the same thing in slightly different words?
The answer, it turns out, is mostly the latter -- and the numbers are stark enough to matter for anyone building LLM-in-the-loop optimization systems.
The trap at the heart of LLM-driven evolution
Evolutionary algorithms rely on the premise that simple variation operators, applied iteratively, can give rise to a vast and diverse space of candidate solutions. Among these operators, mutation plays a central role, continually injecting new genetic material and enabling populations to explore beyond their current configurations. LLMs seemed like a natural upgrade to this process: they can make changes that preserve semantic intent, respect syntax, and even reason about what a program does. LLMs are well-placed to act as selection and mutation operators, as they have been trained on vast datasets of human knowledge and culture, offering a mechanism through which to make semantically meaningful mutations via text.
But there is a catch. Multiple independent LLM ideation sessions frequently converge around similar themes and concepts. This convergence results in a narrow exploration of the ideation space, and the computer science literature broadly refers to this phenomenon as "mode collapse" -- a fundamental limitation of current LLM design. The new paper, Mutation Without Variation, is the first to rigorously quantify this collapse specifically in the context of program evolution, with a controlled experimental setup designed to isolate the mutation operator itself.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
