Qwen's Q-Judger Replaces AI Judges With 80 Human Artists Across 56 Rubrics
Qwen releases Q-Judger, a 27B vision model trained by 80 art-school experts to score AI-generated images across 56 fine-grained dimensions — revealing a hard ceiling no model has broken.
- Qwen releases Q-Judger, a 27B vision-language judge model for evaluating text-to-image generation across 56 fine-grained rubrics.
- Built on Qwen3.6-27B and fine-tuned on 130,000+ human-labeled pairs annotated by 80 art-school professionals under blind, triple-review protocols.
- Evaluates images across 5 pillars: Quality, Aesthetics, Alignment, Real-world Fidelity, and Creative Generation — the last two being novel additions targeting professional creator workflows.
- Benchmarked 18 leading T2I models; GPT Image 2 leads at 64.7 overall, but no model scores above 44 on Physical Logic, Anatomical Fidelity, Animals, Objects, or Contact Interaction.
- Achieves Spearman ρ = 0.92 correlation with human expert rankings, matching professional judgment while enabling fully offline, reproducible evaluation.
- Available free under Apache 2.0 on GitHub with vLLM/SGLang support; companion paper and 1,000-prompt bilingual dataset also released.
Evaluating text-to-image (T2I) models has always been a messy problem. Most benchmarks reduce everything to a single number , CLIP similarity, a preference score, or a coarse human rating , and as frontier models have gotten better, those numbers have compressed into indistinguishable bands. Qwen's research team argues that the field has been measuring the wrong things, and they've built a benchmark and a judge model to prove it.
A judge trained by artists, not by another AI
Q-Judger is a vision-language model fine-tuned specifically for automated evaluation of text-to-image generated images. It is trained on a unified judge model based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols, scoring every image across all 56 verifiable facets and producing fine-grained, rubric-grounded, and fully attributable diagnostics rather than a single opaque score.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
