Skip to main content
Back

Fudan's MulDimIF Trains 8B Models to Match Claude on Multi-Constraint Following

A new benchmark shows top LLMs drop from 80% to 37% accuracy as instruction constraints stack up, and a targeted RL fix nearly closes the gap.

  • MulDimIF benchmarks LLMs across 3 constraint patterns, 4 categories, and 4 difficulty levels using 9,106 code-verifiable samples.
  • Average accuracy collapses from 80.82% at Level I to 36.76% at Level IV across 18 tested models.
  • Best overall scorer is Claude 3.5 Sonnet at 67.50%; scaling up does not always help within a family.
  • GRPO training on 7,906 samples lifts LLaMA3.1-8B from 36% to 88% without hurting MMLU or math.
  • Parameter analysis shows gains come almost entirely from attention module updates across all layers.
  • Code, data and generation pipeline released on GitHub and Hugging Face.

Instruction following sounds simple until you try to measure it properly. When a user asks a model to answer in Markdown, using only capitalized words, in under 50 tokens, ending with a period, models start dropping constraints one by one. A new framework called MulDimIF, accepted to ACL 2026, digs into how badly this scales and offers a training recipe that mostly fixes it.

The team behind it, from Fudan University, Lenovo Research, and Tencent, built a controllable pipeline to generate 9,106 code-verifiable instructions, benchmarked 18 leading models on them, and then used the same data to retrain smaller LLMs with reinforcement learning. The gains are large, and the parameter-level analysis points to a specific culprit inside the model.

Why existing benchmarks miss the point

Benchmarks such as IFEval, FollowEval, and FollowBench evaluate models on dimensions like logical reasoning and stylistic consistency, using either code-based or LLM-based scoring. Most focus on the variety of constraint types (JSON, keyword inclusion, word count) but ignore two axes that matter in real prompts: how constraints are phrased, and how many pile up at once.

MulDimIF organizes the space along three dimensions:

  • Constraint pattern: how constraints are delivered. Example (in-context question-answer pairs sharing the same constraint type), listing (a numbered, point-by-point format), and incorporation (constraints woven directly into the sentence).
  • Constraint category: four buckets (content, format, language, length) split into 13 subcategories like XML attribute counts, Markdown heading levels, or keyword inclusion.