DKPS Cuts AI Benchmark Costs 16x by Reusing Cached Model Responses
A new paper introduces DKPS, a method that reuses cached model responses to predict benchmark scores with far fewer queries than standard evaluation requires.
- Researchers introduce DKPS, a method to predict a new model's benchmark score by reusing cached responses from previously-evaluated models.
- On legalbench, DKPS achieves the same prediction accuracy as direct sampling with 16x fewer queries.
- More broadly, DKPS can reduce the query budget needed to hit a target accuracy by more than 10x.
- An offline query selection strategy lets you pre-compute an optimal question subset before the new model is ever queried.
- The method works in the black-box setting -- no access to model weights or activations required, only text outputs.
- Tested on four HELM-Lite tasks: legalbench, med_qa, wmt_14, and math, using a Leave-One-Family-Out evaluation protocol.
Every time a new model ships, someone has to run the benchmarks. That means sending thousands of prompts to the model, collecting responses, and scoring them -- a process that is often necessary to understand a model's behavior before deployment, but for modern evaluation frameworks, generating and evaluating a response for all queries can be prohibitively expensive. Researchers Hayden Helm, Ben Johnson, and Carey Priebe have published a paper proposing a smarter path: instead of querying a new model on every benchmark item, use the responses already cached from previously-evaluated models to predict how the new one will score.
The cache sitting right under your nose
Benchmark platforms like HELM Lite -- Stanford CRFM's lightweight evaluation suite -- accumulate enormous amounts of model response data over time. In practice, responses from previously-evaluated models are often cached, creating a potential opportunity to use this additional information to decrease the number of queries required to accurately evaluate a new model. The insight is deceptively simple: if you already know how GPT-4o, Llama 3, and a dozen other models answered a set of questions, that collective behavior tells you something about the geometry of the problem space -- and where a new model is likely to land in it.
The challenge is formalizing that intuition into something mathematically rigorous. That is where DKPS comes in.
What is the Data Kernel Perspective Space?
DKPS is a mathematical framework for analyzing the statistical properties of model outputs. It permits quantifying the distance between model inputs and outputs even when the model weights themselves are not accessible -- such as with OpenAI's GPT family. DKPS represents the mean discrepancy geometry for a collection of models from the perspective of an evaluation query set. In plain terms: given a shared set of prompts, DKPS measures how differently each model responds and embeds those differences into a low-dimensional coordinate space. Models that behave similarly end up close together; models that diverge end up far apart.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.