Alibaba's WIMPE Grades Long AI Answers the Way Human Examiners Do
A new framework factorizes reference answers into weighted scoring points, mimicking how human graders assess long-form responses with heterogeneous importance.
ProTakeaways
- WIMPE evaluates long-form LLM outputs by factorizing reference answers into weighted scoring points, inspired by human graders. Paper
- Two metrics: WPA rewards weighted coverage, PCP penalizes contradictions with the reference.
- Points get 3-level importance weights (Critical=3, Moderate=2, Minor=1) and 3-level alignment scores (0, 0.5, 1).
- Beats BLEU, ROUGE-L, Checklist, and coarse rubrics on all 10 long-context benchmarks in Spearman and Kendall correlation.
- Lightweight Qwen2.5-0.5B evaluator matches GPT-4o's ranking correlation, drastically cutting evaluation cost.
- Authors will release weighted scoring points for widely used generative benchmarks. Full HTML paper
Grading a long, multi-part answer is hard. If you ask an LLM to summarize a hotel review, write a legal brief, or answer a scientific question, the expected response usually contains several distinct claims that a good evaluator should check one by one. Yet most automatic metrics still hand out a single holistic score, which makes it difficult to tell whether a model actually covered the important points or just wrote something fluent.
A team from Pengcheng Lab, CUHK, Fudan, and the Qwen team at Alibaba proposes WIMPE (Weighted Importance Multi-Point Evaluation), a framework that grades long-form generations the way a human examiner would: break the reference answer into scoring points, weight each by how essential it is, then score coverage and contradictions separately.
Where current LLM judges fall short
Two evaluation styles dominate the recent literature. Some works employ task-level rubrics for pair-wise comparison or direct scoring, constructing abstract dimensions like correctness or completeness that apply across diverse instances. Others build question-aware checklists with yes/no items. Both have blind spots:
- Task-level rubrics ignore instance-specific facts and can't tell whether an answer is grounded in the provided context.
- Checklists usually derive items from the question alone, not from the reference or context, and treat every item as equally important.