Skip to main content
Back

NYU's BabyCL Learns Words Like a Child Without Shuffling Training Data

BabyCL learns word meanings from a single chronological pass of a child's video, matching offline-trained models without ever reshuffling the data

  • BabyCL is a new continual learning framework that processes a child's egocentric video in a single chronological pass to learn word-referent mappings.
  • Unlike prior models, it never shuffles or repeats data, processing 221 hours of SAYCam footage and 37,486 utterances exactly once in order.
  • A dual replay buffer independently manages visual and multimodal memory to prevent catastrophic forgetting across both objectives.
  • BabyCL outperforms all streaming baselines on the 4AFC word-grounding benchmark, substantially closing the gap to offline-trained upper bounds.
  • Ablations show gains are robust to temporal window size and buffer eviction strategy, suggesting a solid architectural foundation.
  • No public code or pretrained model has been released yet; the paper is on arXiv.

Every child learns language the hard way: one moment at a time, in order, with no rewinds. A neural network trained to do the same thing has historically cheated -- shuffling the data, running hundreds of passes, and treating the problem like a static dataset. A new paper from researchers at NYU challenges that assumption head-on, introducing BabyCL, a continual multimodal learning framework that processes a child's egocentric video in a single, chronological sweep and still learns to match words to the things they refer to.

The gap between how kids learn and how models train

The research builds on a growing body of work using the SAYCam dataset -- head-mounted camera footage from three children aged 6 to 32 months, recording what a developing infant actually sees and hears. SAYCam captures the egocentric, longitudinal perspective of 3 different children aged 6 to 32 months, collected via head-mounted cameras. These datasets provide a unique glimpse into what children actually see and hear over the course of language learning, providing a testbed for what can be learned by training models from this kind of naturalistic, high-quality input.

Prior work showed that neural networks could learn word-referent mappings from this footage -- the ability to connect a spoken word like "ball" to the right visual category. But there was a critical flaw in the setup. These models cycle through the shuffled data for hundreds of epochs, contrasting with how children actually encounter their environment. That's not learning like a child. That's offline batch training wearing a developmental costume.

The core problem is called catastrophic forgetting: when a neural network learns new information sequentially, gradient updates from new data tend to overwrite what was learned from older data. In lifelong and continual learning, replay buffers are crucial for mitigating catastrophic forgetting. Without a solution to this, streaming learning on temporally ordered video degrades quickly.

What BabyCL actually does