Qwen3-VL Beats Traditional Image Quality Scores With Expert-Level Visual Reasoning
A new benchmark asks multimodal models to pick the sharper photo from expert-grade image pairs and explain their reasoning like a pro.
Takeaways
- NTIRE 2026 RAIM Track 1 benchmarks multimodal LLMs on choosing the better of two high-quality photos and explaining why.
- Dataset covers 100 train, 102 validation, 101 test image pairs, expert-annotated across portraits and scenery.
- Evaluation combines binary accuracy with BLEU-4/ROUGE-L reasoning score and an LLM-as-a-Judge semantic check.
- 192 registrations produced 2,530 Phase 2 submissions; Team IH-VQA won with a 0.7305 final award score.
- Every top method fine-tuned Qwen3-VL-8B-Instruct with GRPO plus multi-checkpoint voting ensembles.
- Dataset and baseline are open at GitHub; leaderboard on CodaBench.
Traditional image quality metrics have a fundamental problem: they hand you a single number. That works when comparing a JPEG artifact to a clean photo, but it collapses when both images are already excellent, which is exactly the situation professional photographers and camera engineers face every day. The NTIRE 2026 RAIM Track 1 challenge report lays out a benchmark that pushes past scalar scores entirely, asking multimodal LLMs to compare high-quality image pairs and justify their choice with expert-grade reasoning.
Why one number falls apart
Conventional IQA methods regress complex visual characteristics into scalar Mean Opinion Scores based on low-level statistics or extracted features. Compressing rich visual information into such a narrow metric leaves these approaches struggling to distinguish subtle differences among uniformly high-quality images. A solitary score also fails to articulate why one image is perceived as superior to another.
That gap matters when tuning a smartphone camera pipeline or picking between two takes of a portrait. You don't need a 7.3 vs 7.4; you need to know that the skin texture on image A is smoother while image B retains more hair detail. The challenge treats this as a job for multimodal LLMs, which couple visual perception with linguistic generation and can explicitly leverage features like textures, noise, and artifacts to formulate grounded reasoning.