Moonshot AI's PerceptionBench Reveals No Frontier Model Sees Past 60%

When a multimodal model gets a visual question wrong, is it because it failed to see something, or because it failed to reason about it? That distinction sounds simple, but almost no existing benchmark actually separates the two. PerceptionBench, released by Moonshot AI, is built specifically to answer that question, and the results are uncomfortable.

The Gap in Visual Benchmarking

Most visual benchmarks test end-to-end performance: give a model an image and a question, then score the answer. A wrong answer could mean the model didn't see the object, misread a spatial relationship, hallucinated something absent, or failed to reason through the problem. These are distinct failure modes, and conflating them makes it nearly impossible to know what to fix.

Holistic evaluations conflate perceptual errors with reasoning failures, while application-driven benchmarks cover only narrow, fragmented domains. PerceptionBench takes a different approach entirely.

Built From Actual Model Failures

Rather than deciding upfront what "visual perception" means, the Moonshot AI team looked at where frontier models actually break. By tracing failures across 42 existing benchmarks to their earliest visual cause, they distilled 10 perceptual capabilities and 3,000 verified questions, each answerable by looking, with no reasoning or outside knowledge required. That bottom-up, failure-driven taxonomy is the core methodological contribution.

The 10 atomic categories are:

• Visual Relation — spatial and structural relationships between objects
• Counting — enumerating objects in a scene
• Attribute — identifying properties like color, shape, and size
• Depth & 3D — understanding spatial depth and 3D structure
• Localization — pinpointing where something appears in an image
• Comparison — judging differences or similarities between visual elements
• Fine-grained Recognition — distinguishing visually similar objects or details
• Context Integration — combining multiple visual cues across a scene
• OCR — reading text embedded in images
• Hallucination — correctly identifying when something is not present

The 3,000 questions are subsampled with capability-level balancing and difficulty stratification from an in-house pool of 17,000+ verified samples. Of those, 1,800 (60%) are atomic sub-questions decomposed from attributed failures on source benchmarks; the remaining 1,200 (40%) are newly authored on supplemented images.

No Model Clears 60%

The headline finding is blunt: no evaluated model exceeds 60% accuracy. Sixteen frontier MLLMs (ten proprietary, six open-source) were tested with unified prompts and the highest available reasoning budget. All questions are open-ended with short, uniquely determined answers; a GPT-oss-120B judge scores each response against the reference, achieving 99.7% agreement with human judgment on a 300-sample audit.

#	Model	Overall	Localization	OCR	Hallucination
1	GPT-5.6-Sol	59.7	76.7	54.9	26.9
2	Kimi K3	58.5	70.3	61.2	41.7
3	Claude-Fable-5	57.2	70.4	64.3	45.0
4	Gemini-3.1-Pro	56.2	52.7	64.3	40.6
5	GPT-5.5	55.8	65.8	56.5	34.7
8	Qwen3.7-Plus	51.1	52.7	54.5	29.5
12	Grok-4.5	41.0	39.7	43.9	44.7

Two models with nearly identical overall scores can show completely different perceptual strengths and weaknesses, a split that aggregate scores hide entirely. The hallucination column makes this vivid: GPT-5.6-Sol, the top overall performer, scores only 26.9% on hallucination, meaning it frequently "sees" things that aren't there. Gemini-3.5-Flash, sitting at 52% overall, scores 50.6% on hallucination. These are fundamentally different failure profiles, invisible on a single leaderboard.

Guessing vs. Perceiving

A large share of correct answers fail to survive a repeated ask, which suggests models are often pattern-matching rather than perceiving. A model that answers correctly 60% of the time but gives a different answer on a second pass isn't perceiving reliably. PerceptionBench is designed to surface exactly this kind of fragile performance.

The benchmark also exposes a structural problem in the evaluation landscape. Each source benchmark captures a narrow slice of perception errors, and those slices overlap only weakly (mean pairwise weighted Jaccard: 0.20). No single benchmark, or small group of them, covers perception as a whole, so combining a handful of existing evals won't approximate what PerceptionBench measures.

Running It on Your Own Model

The dataset is available under CC-BY-NC-4.0 on Hugging Face, and the evaluation code is on GitHub under Apache 2.0. The eval harness accepts any OpenAI-compatible endpoint:

git clone [email protected]:MoonshotAI/PerceptionBench.git
cd PerceptionBench
pip install -r requirements.txt
export OPENAI_API_KEY="your-api-key"
export OPENAI_BASE_URL="https://your-endpoint/v1"
export MODEL="your-model-name"
python eval/eval.py

Results are written per-item to results/PerceptionBench_<model>.jsonl and per-category accuracy to results/PerceptionBench_<model>_scores.json. The judge model defaults to gpt-oss-120b and is configurable via the JUDGE_MODEL environment variable.

What This Changes for Builders

Strong performance on holistic visual benchmarks like VQA or MMBench doesn't necessarily reflect strong visual perception. A model can score well on those by compensating for weak perception with strong language priors, essentially inferring what an image probably contains from context rather than actually reading it.

For teams building multimodal systems, PerceptionBench works best as a diagnostic. A model scoring 55% overall but only 30% on depth perception and 35% on fine-grained recognition tells you exactly where to invest in training data or architectural changes. That kind of targeted signal has been largely absent from the evaluation toolkit until now.