Skip to main content
Back

Oregon State's EvoPool Beats GPT-4o Annotation at 4,500x Less Cost

EvoPool uses Darwinian evolution to breed Python annotation functions that label 100K examples 31,000x faster than GPT-4o while beating LLM annotation on 7 of 8 specialized tasks

  • EvoPool breeds executable Python annotators through Darwinian evolution, beating LLM annotation on 7 of 8 specialized tasks by an average +0.141 macro-F1.
  • Once authored, the pool annotates 100K examples in 17.2 seconds at zero cost — up to 31,000x faster than GPT-4o-mini API calls.
  • Three specialized agents (Generator, Improver, Refiner) propose annotators each generation; a deterministic gate enforces precision, diversity, and marginal contribution.
  • EvoAgg, a text-aware aggregator combining sentence-BERT embeddings with annotator votes, adds up to +0.254 macro-F1 over majority vote on top of pool evolution gains.
  • Largest gains on ChemProt (+0.301), PubMed (+0.265), and DDI (+0.219); SciFact is the one loss where GPT-4o-mini's scientific pretraining wins.
  • Code and annotator pools are open-source at github.com/tianyi0216/EvoPool; paper at arxiv.org/abs/2606.01617.

Getting training labels for specialized domains is brutally expensive. Biomedical relation extraction, legal clause classification, scientific claim verification , these tasks require clinicians, lawyers, or domain reviewers, and the cost routinely runs into the hundreds of thousands of dollars per dataset. The standard workaround, using a large language model to label your training data, sounds appealing but falls apart in practice: on the ChemProt biomedical benchmark, GPT-4o-mini annotation achieves only 0.28 macro-F1. A paper published on arxiv by Tianyi Xu, Yaolun Zhang, Xuan Ouyang, and Huazheng Wang from Oregon State University and the University of Wisconsin-Madison proposes a different approach: instead of asking an LLM to label examples one by one, evolve a pool of executable Python annotators that can do it for free.

The problem with LLM annotation at scale

There are two natural ways to use LLMs for data labeling. The first is direct annotation: send each training example to the LLM and collect its label. This works reasonably well on general tasks but fails on specialized domains where the LLM lacks deep expertise. It also costs money and time that scale linearly with your corpus , annotating 100K examples with GPT-4o-mini takes 21.9 hours and $2.95. The second approach, used by tools like Alchemist and DataSculpt, asks the LLM to write a pool of executable labeling functions (short Python programs that classify text) in a single shot. These functions run cheaply at inference, but they inherit whatever biases the LLM had before seeing your data. They collapse on rare classes and never adapt to what the model actually gets wrong.

EvoPool attacks both failure modes at once. The core idea is to treat the pool of annotators as a population that evolves under selection pressure from a small labeled validation set , the only ground-truth labels the system ever sees.

Darwinian annotation: how it works

The paper frames annotation as an evolutionary search problem. Three role-specialized LLM agents act as mutation operators, and a deterministic selection gate acts as natural selection. Each generation, the agents propose new Python annotator functions; the gate keeps only those that pass three filters: