Skip to main content
Back

NVIDIA's LocateAnything Detects Objects 10x Faster by Ditching Sequential Decoding

NVIDIA's LocateAnything uses Parallel Box Decoding to predict bounding boxes 10x faster than text-based VLMs, without sacrificing accuracy

  • Parallel Box Decoding: LocateAnything predicts all four bounding box coordinates simultaneously instead of token-by-token, improving both speed and geometric accuracy.
  • 10x throughput gain: Achieves 12.7 boxes/sec vs. 1.1 BPS for Qwen3-VL; 2.5x faster than quantized competitor Rex-Omni.
  • SOTA across 6 task types: Tops benchmarks in GUI grounding (ScreenSpot-Pro), document layout (DocLayNet, M6Doc), dense detection (LVIS, VisDrone), and OCR.
  • Massive training corpus: Trained on 138M queries and 785M bounding boxes across 12M images spanning photos, GUIs, documents, and more.
  • Hybrid inference: Single model supports fast parallel mode, slow autoregressive mode, and a hybrid fallback for ambiguous scenes -- no retraining needed.
  • Available now (non-commercial): Model on HuggingFace, live demo, and code on GitHub; dataset release pending.

Most vision-language models treat object detection as a writing exercise. They serialize a bounding box into a string of coordinate tokens -- x1, then y1, then x2, then y2 -- and generate them one at a time, the same way a language model writes a sentence. It works, but it is slow and geometrically naive: the model has no built-in understanding that those four numbers belong together as a rectangle. LocateAnything, a CVPR 2026 paper from NVIDIA Research, fixes this with a fundamentally different approach that hit #1 trending on Hugging Face the day it dropped.

The bottleneck nobody talks about

When you deploy a vision-language model for detection in a real system -- a robot arm, a GUI agent, an autonomous vehicle -- the bottleneck is rarely the vision encoder. It is the autoregressive decoder grinding out coordinate tokens sequentially. VLMs commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This creates two compounding problems: token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation.

The geometry problem is subtle but important. When a model predicts x1 without knowing x2 yet, it has no way to enforce that the resulting box makes spatial sense. Coordinates end up inconsistent, especially in dense scenes where objects overlap. Prior work tried quantizing coordinates into discrete bins ("quantized coordinate decoding") to speed things up, but this only partially addresses the sequential bottleneck and introduces its own accuracy tradeoffs.

Boxes as atomic units

LocateAnything's core innovation, Parallel Box Decoding (PBD), predicts complete bounding box coordinates in a single parallel step rather than autoregressive token-by-token decoding, improving efficiency while preserving geometric consistency. Think of it like the difference between writing a sentence word-by-word versus stamping the whole word at once. Each box -- all four coordinates together -- is treated as one indivisible prediction unit.

Comparison of standard token decoding vs Parallel Box Decoding

The architecture is built on a Moon-ViT vision encoder and a Qwen2.5 language decoder, bridged by an MLP projector, directly converting visual tokens into a sequence of box-aligned block-level predictions. The output is organized into fixed-length blocks of length 6: a semantic label block, a box coordinate block, a negative block, and an end block. Coordinates are normalized integers in the range [0, 1000], making them resolution-agnostic.

Three modes, one model

A key engineering insight is that parallel decoding is not always the right call. Dense scenes with heavily overlapping objects can create "spatial ambiguity" -- the model is uncertain which coordinate belongs to which object. LocateAnything handles this with a hybrid inference strategy:

  • Fast Mode (MTP): All boxes predicted in parallel. Maximum throughput, best for latency-constrained settings like robotics and on-device agents.
  • Slow Mode (NTP): Standard autoregressive decoding. Most robust, best for high-precision offline labeling.
  • Hybrid Mode (default): Starts in Fast Mode, detects when a box prediction is unreliable (format irregularity or spatial ambiguity), discards that block, and falls back to Slow Mode for just that box before resuming parallel decoding.

The architecture jointly optimizes next-token prediction and multi-token prediction to balance reasoning ability and parallel inference. Training follows a four-stage pipeline: initial multimodal knowledge adaptation using captioning, VQA, OCR, and related data, followed by grounding and dense-scene localization fine-tuning.

Architecture diagram of LocateAnything with Moon-ViT encoder, MLP projector, and Qwen2.5 LLM

138 million queries, 785 million boxes

The model's breadth comes from its training data. LocateAnything-Data is a multi-domain dataset encompassing 12M unique images and massive, dense supervisory spatial signals. The query breakdown reveals the intentional design for agentic use cases:

  • General object detection: 66.9% of queries, 83.1% of boxes -- the dense localization backbone
  • GUI element grounding: 16.5% of queries -- for agent navigation of app interfaces
  • Referring comprehension: 7.3% -- linking natural language to specific regions
  • Text localization (OCR): 3.6% -- tight grounding of text in images
  • Layout grounding: 3.5% -- document and scene structure understanding
  • Point-based localization: 2.2% -- fine-grained coordinate precision

Annotation was generated using a hybrid pipeline combining human labels, open-source datasets, and model-assisted synthetic annotations from Qwen3-VL, Molmo, SAM 3, and Rex-Omni, with automated post-verification.

The numbers that matter

Speed is the headline, but accuracy is what makes it deployable. LocateAnything achieves 12.7 BPS (boxes per second), over 10x faster than textual-based Qwen3-VL (1.1 BPS) and 2.5x faster than quantized-based Rex-Omni (5.0 BPS). Critically, this speed does not come at an accuracy cost -- it comes with accuracy gains.

On the benchmarks that matter most for real deployments:

  • LVIS (long-tail detection): +3.8% mean F1 improvement over Rex-Omni at identical model size, with particularly strong gains at high IoU thresholds (31.1 vs. 20.7 at IoU=0.95).
  • ScreenSpot-Pro (GUI grounding): SOTA mean F1 of 60.3, surpassing generalist VLMs like Qwen3-VL-30B-A3B and specialized models such as GUI-Owl-32B.
  • Document understanding: 76.8 and 70.1 mean F1 on DocLayNet and M6Doc respectively, outperforming Rex-Omni by substantial margins (+6.1 / +14.5).
  • Dense scenes: On Dense200 and VisDrone, LocateAnything achieves 58.7 and 39.9 mean F1 respectively, demonstrating superior boundary delineation in heavily overlapping environments.

The high-IoU gains (IoU=0.95) are particularly significant. This threshold requires near-pixel-perfect box placement -- the kind of precision that matters for robotic manipulation, where a few pixels of error can mean a missed grasp. The geometric coherence of predicting all four coordinates together is directly responsible for this improvement.

ScreenSpot-Pro GUI grounding benchmark results comparing LocateAnything against other models

Where it fits -- and where it does not

LocateAnything enables precise object localization, dense detection, and point-based localization across diverse domains in both Enterprise Intelligence and Physical AI, supporting tasks such as referring expression grounding, multi-object detection, GUI element grounding, and text localization, with strong performance in complex and cluttered scenes. It has already been integrated into NVIDIA's Nemotron 3 Nano Omni production model.

The practical use-cases break into two families:

  • Agentic systems: GUI automation, computer-use agents, web scraping with visual grounding, robotic manipulation pipelines
  • Data pipelines: Automated dataset labeling, document layout parsing, OCR localization at scale, industrial inspection

The main constraint to know about: the model is released under an NVIDIA non-commercial license. Commercial use is not permitted except by NVIDIA and its affiliates, so it is research and prototyping only for most teams. The dataset is also listed as "incoming" -- not yet publicly released at time of writing.

Getting started

The LocateAnything-3B model is available on Hugging Face, with a live demo space and code in the NVlabs/Eagle GitHub repo. The simplest way to run it:

from transformers import AutoModel, AutoTokenizer, AutoProcessor
from PIL import Image
model = AutoModel.from_pretrained(
    "nvidia/LocateAnything-3B",
    torch_dtype="auto",
    trust_remote_code=True
).cuda().eval()
# Detect objects with a text query
# Output: structured XML-like token sequence with  coordinates
# e.g. <142><310><489><756>

The model supports generation_mode="hybrid" (default), "fast", and "slow" at inference time -- no retraining needed to switch between them. For teams building perception pipelines where throughput is a bottleneck, this is worth benchmarking against whatever specialized detector you are currently running.