VisionHOPE lets a visual backbone adapt how it learns

VisionHOPE is a newly released open-source visual backbone that changes its internal update controls while processing each image. Its authors published full PyTorch code, pretrained weights, and training recipes for ImageNet-1K, COCO, and ADE20K in a GitHub repository under the MIT license. The package includes Tiny, Small, and Base models alongside reusable operator modules.

The design represents its update process through five temporary memories covering content, keys, values, learning rate, and retention. These states evolve during a forward pass. Here, “learning” means updating temporary within-image memory; gradient-based checkpoint training remains a separate process.

Update rules become state

Modern visual architectures use several forms of input-conditioned computation. Convolutional Neural Networks aggregate local neighborhoods, Vision Transformers build input-dependent attention maps, State-Space Models vary state transitions, and Test-Time Training layers update an inner learner while processing an example.

VisionHOPE draws on Nested Learning, a framework that treats a model as coupled learning processes. Its temporary content memory changes alongside the variables that control how strongly new information enters and how much previous information remains.

A Transformer keeps its learned projection matrices fixed during inference even though its attention maps depend on the input. VisionHOPE extends input conditioning to optimizer-like controls inside the operator, allowing update size and retention to change token by token.

Five memories move together

During each scan, the operator maintains five coupled states with distinct roles: