Skip to main content
Back

PACO Fixes the Category Explosion Flaw Crippling Vision AI Streams

A new inference-time framework replaces the single-threshold decision in streaming category discovery with calibrated, hierarchical decisions that adapt as data arrives.

Pro

  • PACO reframes On-the-Fly Category Discovery as a hierarchical decision tree instead of a single global threshold.
  • Three stages route each streaming sample: known-class evidence, birth-aware novel assignment, and attach-versus-create.
  • Decision thresholds are calibrated from support-set proxy tasks and refined online with mature novel prototypes.
  • Achieves 57.7 percent average Strict-Hungarian accuracy across seven OCD benchmarks, versus 45.9 for AGE.
  • Fixes category explosion: 194 predicted clusters on CUB versus 2146 for SMILE-32bit at a true count of 200.
  • Plug-and-play module that improves AGE and PHE-12bit without retraining or dataset-specific tuning.

When a vision model sees a picture of an animal it has never encountered, it should ideally do three things at once: decide the sample is not one of the classes it already knows, notice if the same novel category has been seen before in the stream, and, if not, spin up a fresh cluster to hold it. This setting is known as On-the-Fly Category Discovery (OCD), and a new paper called PACO argues that the field has been solving the wrong half of the problem.

The bottleneck nobody wanted to fix

OCD makes a model instantaneously aware of novel category samples, enabling inductive learning and streaming inference from an unlabeled test feed. Recent work has made strong progress on the representation side: SMILE uses hash-based descriptors, PHE improves them with prototypical hashes, DiffGRE synthesizes pseudo-novel samples, and AGE models category densities with adaptive Gaussians.

PACO's authors point out that all of these systems compress the actual online discovery step into a single threshold separating known from novel. OCD, they argue, is a sequence of coupled decisions, and squashing them into one rule makes the system fragile in fine-grained settings where inter-class gaps are small. Early errors propagate through the stream, causing what the authors call category explosion.

How bad is the explosion, really?

The numbers make the case bluntly. On Stanford Cars, which has 196 ground-truth classes, SMILE-32bit predicts 2953 clusters. PHE-32bit still fragments the space into 762 pseudo-classes. Even the more recent AGE ends up at 633. PACO predicts 367. On CUB-200-2011, PACO estimates 194 categories against a true count of 200, while SMILE-32bit hallucinates 2146.