CMU Argues Static AI Benchmarks Fail When Models Actually Compete
Researchers from CMU and UMass Amherst argue static benchmarks miss competitive effects, and release a simulation framework where AI agents compete for users over time.
Takeaways
- CMU and UMass researchers propose Marketplace Evaluation, treating AI agents as competitors in a simulated market.
- Static benchmarks miss switching, routing, and early-adoption effects that determine real deployment outcomes.
- Motivating experiment with seven models on SimpleQA shows simulation rankings diverge from static F1 rankings.
- Framework covers users, generators, routers, and retrievers with evolving preferences via agent-based simulation.
- New metrics include windowed market share, customer retention, and the Herfindahl-Hirschman concentration index.
- Open-source Python package marketplace-eval ships with the motivating experiment reproducible out of the box.
A team from Carnegie Mellon and UMass Amherst has published a paper arguing that the way we evaluate retrieval systems and language model agents is misaligned with how they actually get deployed. Their proposal, called Marketplace Evaluation, treats information access systems as competitors fighting for users inside a simulated market instead of scoring them one at a time on a fixed test set.
Leaderboard position, the authors argue, does not translate to market success. Once a model ships into a live marketplace like OpenRouter or an enterprise catalog, its fate depends on user switching behavior, routing decisions, competitor moves, and early-adoption effects that no static benchmark can capture. The paper formalizes a framework, runs a motivating experiment to show the effect exists, and releases an open-source Python package so others can build on it.
Where isolated benchmarks break down
Evaluation of IA agents is dominated by static benchmarks that treat systems as standalone artifacts, and real deployment looks nothing like that. In the Cranfield paradigm, evaluation can be read as a multi-user single-system setting, where many queries stand in for many users but all interactions are routed to a fixed system, eliminating any notion of competition or user choice. Pairwise and Arena-style setups extend this to a multi-user multi-system version, yet still fall short of modeling real competition: users are typically presented with anonymous variants and do not actively select among providers, preventing the formation of persistent preferences or switching behavior.
Deployment involves winner-take-all effects, where small performance gaps can produce dramatic market share disparities, along with portfolio effects, where generators may diversify across multiple retrieval systems to handle different query types or hedge against failures. Miss those dynamics and you cannot predict which system will actually get adopted.