Ohio State Exposes How DiagHOI Reveals AI's Blind Spots in Crowded Scenes
Researchers dissect where popular two-stage human-object interaction detectors quietly fail, revealing that high benchmark scores mask serious reasoning gaps in crowded scenes.
Takeaways
- Study dissects failure modes of four two-stage HOI detectors beyond aggregate mAP scores.
- HICO-DET is reorganized into SPSO, SPMO and six multi-person categories A through F.
- Every model loses mAP moving from single-person to multi-person scenes, with category C worst.
- Verb prediction errors dominate and persist even at confidence above 0.5.
- HOLa and LAIN have fewer pairing errors thanks to instance-level feature representations.
- Models show object-conditioned bias, favoring verbs that dominate each object's training distribution.
Human-object interaction (HOI) detection has been climbing steadily on benchmarks for years, but a new diagnostic study argues that the scoreboard is misleading. A team from Ohio State, NUS, Boston University and collaborators took four popular two-stage HOI detectors, ran them through a carefully sliced version of the standard test set, and found that the models fall apart in exactly the scenes that matter most for real applications: crowded rooms, shared objects, and rare verb-object combinations.
The paper does not propose a new architecture. It reorganizes the HICO-DET test set into structured subsets so that different failure modes can be isolated and measured, something the standard mean Average Precision (mAP) score cannot do on its own.
Why one number hides a lot of bugs
HOI detection means finding triplets like ride-bicycle or hold-cup in an image. The field has largely converged on two paradigms: one-stage models fold detection and interaction classification into a single pipeline, while two-stage models first detect objects and then classify the interactions between them. This study focuses on the two-stage family because the separation makes it easier to attribute errors to specific components.
The problem the authors flag is structural. Standard HICO-DET evaluation is dominated by simple single-person images, so aggregate mAP tells you very little about whether a model can reason about who is doing what to which object when several people share a scene. Prior diagnostic work has already hinted that across both paradigms, most errors trace to two sources: incorrect localization of the object in a human-object pair, and incorrect interaction classification even when localization is right. This paper pushes further by breaking failures down by scene composition.