KINA Benchmark Exposes Frontier Models Knowing Only Half of Human Knowledge
KINA spans 261 academic disciplines and exposes a tiered frontier where even the best model, Gemini, scores just 53% -- leaving massive room for improvement
Takeaways
- KINA benchmark released: 899 questions across 261 academic disciplines, evaluating 42 models from 13 labs on broad knowledge breadth.
- Frontier models far from saturation: Best score is 53.17% (Gemini-2.1-Pro-Preview) against a 10% random baseline -- enormous headroom remains.
- Tiered leaderboard structure: A small frontier tier above 48% (Gemini, Claude Opus 4.6, GPT-5.4), a dense middle tier at 38-45%, and a long tail near chance.
- Formal proofs for coverage and annotation quality: The paper proves a greedy discipline-selection algorithm achieves (1-1/e) coverage and that bonus-based annotation payment dominates flat pay.
- Tool augmentation adds up to 5.17 points, but gains vary significantly across model families.
- Code and dataset are open-source on GitHub, compatible with any OpenAI-compatible API including local vLLM/SGLang servers.
Benchmarks for large language models are in a quiet crisis. MMLU, once the gold standard of general knowledge evaluation, is now saturated -- frontier models score between 88% and 94%, a range where differences could easily be noise rather than signal. The field has been patching this with harder variants, but MMLU-Pro is itself approaching saturation at the frontier, repeating the very dynamic it was built to fix. A new paper tackles this problem from a different angle entirely: instead of just making questions harder, it asks whether benchmarks are even measuring the right things in the first place.
The paper introduces KINA (Knowledge Index of Noah's Ark), an 899-item benchmark built across 261 fine-grained academic disciplines. The name is a nod to comprehensiveness -- the goal is to preserve a representative sample of human knowledge, not just the domains that happen to be easy to annotate at scale.
Three problems with how we build benchmarks today
The paper identifies three structural flaws in how knowledge benchmarks are typically built:
- Scaling-driven design: Most benchmarks grow by adding more questions in already-covered areas, not by deliberately seeking out underrepresented disciplines. The result is a dataset that looks broad but is actually skewed toward whatever is easiest to source at scale.
- Flat-payment annotation: When annotators are paid a fixed rate regardless of quality, the rational move is to agree with each other quickly -- a phenomenon the authors call "lazy consensus." This inflates apparent inter-annotator agreement without improving actual question quality.
- Unaudited ranking instability: With a fixed test budget, the ranking of models near each other on a leaderboard can flip due to statistical noise alone. Most benchmarks report point estimates without acknowledging this variance.