RelateAnything generates scene graphs at video speed

Scene graphs describe objects in an image and the relationships between them. Existing systems often rely on small, fixed predicate sets and expensive inference. The independent RelateAnything repository instead accepts an image, bounding boxes, and a runtime list of free-text predicates, then returns ranked relationships such as riding, reflected in, or about to collide with.

The release includes a 53.2-million-parameter default model built on DINOv3, a 4.3-million-relation training corpus called RA-4M, and a six-axis evaluation suite named OV-SGG-Bench. The repository publishes Apache-licensed code and DINOv3-based checkpoints. A fully client-side WebGPU/WASM demo processes images in the browser without uploading them.

A runtime vocabulary changes the contract

The project page defines a compact interface: supply an image, boxes from a detector or segmenter, and the predicates the application needs. The model returns directed triplets containing subject and object box indexes, a predicate, and a confidence score.

from relsgg import RelateAnything

model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus")
model.set_vocabulary([
    "about to collide with",
    "reflected in",
])
relations = model.predict(image, boxes)

• Runtime predicates: The caller can replace the vocabulary with arbitrary strings. Updating it requires one pass through a small text encoder and no model retraining.
• Constrained output: Every emitted predicate comes from the supplied list, which gives downstream controllers and rule engines a fixed output schema.
• Class-agnostic regions: Object labels never enter the relation model, so boxes can come from a conventional detector, an open-vocabulary detector, or a class-agnostic segmenter.
• Parallel graph types: One forward pass can produce a single ranked list or separate spatial and semantic graphs. A pair may therefore carry both a layout relation and an interaction.

Where the 20-millisecond figure comes from

The benchmark configuration processes images at 448 pixels and scores up to 128 sampled pairs drawn from 20 boxes using