SSM Framework Makes Facial AUs and Expressions Finally Train Each Other
A new framework called Structured Semantic Mapping lets action unit detection and facial expression recognition train together across mismatched datasets, boosting both.
ProTakeaways
- Structured Semantic Mapping enables bidirectional AU and expression learning across heterogeneous datasets without needing multi-label annotations.
- Three components: shared CLIP-ViT-B/16 visual backbone with MoE layers, Textual Semantic Prototypes, and Dynamic Prior Mapping matrices.
- Reaches 68.5 F1 on BP4D and 71.9 F1 on DISFA, +2.3 over single-task training.
- Hits 75.37 WAR on DFEW, matching or beating recent CLIP-based and self-supervised expression models.
- Code and models available at github.com/MSA-LMC/SSM.
- Cross-domain zero-shot tests show +20.6 F1 gain on DISFA over single-task baselines.
Reading a face is really two problems stacked on top of each other. There are the tiny muscle twitches, called Action Units (AUs), that anatomists catalog under the Facial Action Coding System, and there are the coarse labels humans slap on top, like happy, sad, or fearful. A team from Hefei University of Technology, Southeast University, and the Chinese Academy of Sciences argues these two tasks should be helping each other far more than they currently do, and they have published a paper proposing a framework that finally lets that happen across mismatched datasets.
Their method, called Structured Semantic Mapping (SSM), hits state of the art on both tasks at once. Code and models are available on GitHub.
The awkward relationship between AUs and expressions
Facial expressions exhibit a certain degree of universality across cultures while being driven by coordinated muscle movements, namely Action Units. The Facial Action Coding System (FACS) provides an anatomically grounded description linking AUs to expressions. In principle, knowing that the cheeks are raised and lip corners pulled up (AU6 plus AU12) should help a model conclude a face looks happy, and vice versa.
In practice, research has been lopsided. The prevailing paradigm focuses on unidirectional facilitation, using AU features or statistics as auxiliary signals to improve expression recognition. Going the other direction, using expression labels to help AU detection, has barely been explored. The reason is boring but decisive: the datasets do not line up.

AU datasets like BP4D and DISFA are shot in laboratories with frame-level annotations by trained FACS coders. Expression datasets like DFEW, MAFW, and FERV39K are scraped from movies and labeled at the clip level with a single emotion tag. Different environments, different annotation granularities, different label spaces. Frame-level multi-label annotation requires professional coders working video by video, which is costly, slow, and hard to scale.