Smarter Image Labeling Boosts ResNet Accuracy by 23% Without Changing the Model
A hierarchy-guided crowdsourcing framework that forces annotators to verify visual properties rather than pick a name, boosting agreement to 0.974.
Takeaways
- New crowdsourcing method labels images by visual properties, not category names, cutting annotator subjectivity.
- Annotators walk a category tree, verifying visual genus and differentia at each node.
- Krippendorff's Alpha rises from 0.912 (names only) to 0.974 with full visual-property protocol.
- All eight tested CNN backbones improve when trained on the new labels; ResNet gains 23.44 percent.
- Ablation shows visual properties matter more than hierarchy alone, adding +0.04 Alpha.
- Optimal task size is 50 images per annotator; extra time cost per task is under one minute.
Image datasets are the fuel for computer vision, yet the way we label them has barely changed since ImageNet: show an annotator a picture, hand them a list of words, ask them to pick one. A new paper argues that this workflow quietly poisons the training signal, and proposes a crowdsourcing protocol that asks annotators about visual properties instead of names.
The bear that is also a teddy, a cartoon, and a costume
The paper opens with two uncomfortable examples from popular benchmarks. In ImageNet, a single guitar image sits simultaneously under "Musical Instrument", "Guitar", and "Acoustic Guitar". In the Open Images Dataset, the "Brown Bear" category lumps together a real animal, a teddy bear (also tagged "Teddy Bear"), a stick-figure cartoon bear, and a person in a bear costume.
The authors trace both failures to the semantic gap problem: the mismatch between what can be extracted from pixels and how a human interprets those pixels in context. When annotators are handed only a word like "bear," the granularity of the label and the criteria for membership are left to personal taste. Three annotators produce three different label distributions, and a downstream classifier inherits the noise.
Visual genus and visual differentia
The proposed methodology borrows a trick from classical taxonomy. Every category in a hierarchy is defined by two things: a visual genus, the properties it shares with its parent, and a visual differentia, the properties that separate it from its siblings. For "Goldfinch," the visual genus is "Finch" (used as the parent category), and the visual differentia is "Crimson face and yellow-and-black wings," which distinguishes it from sibling categories.