NVIDIA's Metropolis VSS 3 Lets AI Agents Deploy Video Search Without Writing Code
NVIDIA's VSS 3 lets coding agents deploy, configure, and query a full video analytics stack through plain-language chat, no microservice wiring required.
Takeaways
- VSS 3 ships agent skills: NVIDIA's Metropolis Video Search and Summarization blueprint now includes modular skills that let coding agents deploy and operate the full stack via chat.
- No manual microservice config: Agents like Codex, Claude Code, and OpenClaw can self-install skills and deploy Docker Compose profiles without hand-written integration code.
- Fusion search is the core upgrade: Multi-embedding search with a VLM critic agent replaces single-vector retrieval, enabling complex multi-attribute queries over video archives.
- Strong benchmark numbers: Single RTX PRO 6000 handles 51 concurrent streams at 1.87s retrieval latency; 1-hour video summarized in under 1 minute.
- Five modular profiles: Base Q&A, alert verification, real-time VLM alerts, agentic search, and long video summarization — all layered on a common Docker Compose base.
- Available now on GitHub: Skills are open on the VSS repo; quickest start is via NVIDIA Brev Launchable with an NGC API key.
Video surveillance and analytics have always had the same painful bottleneck: the footage is there, but getting answers out of it requires a team of engineers to wire together detection models, embedding pipelines, vector databases, and query interfaces. NVIDIA just took a sledgehammer to that bottleneck with a major update to its Metropolis Blueprint for Video Search and Summarization (VSS), version 3, introducing a set of agent skills that let a coding agent handle the entire deployment and query workflow through a chat interface.
The old way was painful
In the past, developers had to manually configure, deploy, and integrate the rich set of microservices VSS provides for video management, search, summarization, and more to build video analytic applications. That meant touching Docker Compose files, configuring NIM microservices, standing up vector databases, and stitching REST endpoints together by hand. For most teams, that friction alone killed adoption before a single frame was ever analyzed.
The core problem VSS is solving is also genuinely hard. Large-scale video search remains one of the most challenging frontiers in modern information retrieval. User queries are inherently complex and ambiguous, and capturing full semantic intent within a single visual embedding is fundamentally insufficient, particularly when objects and events carry multi-layered attributes that resist simple vector representation.
Skills: the agent interface that changes everything
Agent Skills are reusable, self-contained capabilities that follow the agentskills.io specification and package the prompts, reference data, and helper scripts a coding agent needs to operate a deployed VSS Blueprint. Each skill maps a developer intent , "deploy VSS for video search", "add a camera", "summarize this video", "verify these alerts" , onto the corresponding VSS REST, VA-MCP, and VIOS calls.
Skills are versioned alongside the blueprint in the VSS repository, exercised by a CI eval workflow on every change, and consumed by any compatible coding agent such as Claude Code or Codex at either deployment time or runtime. The key design choice: because every skill follows the agentskills.io specification, any compatible harness can load the same skills folder without bespoke integration code.
The available skills in the VSS GitHub repository cover the full operational surface:
- Deploy, debug, or tear down any VSS profile using a Docker Compose-centric workflow.
- Produce video analysis reports by querying the VSS agent's
/generateendpoint. - Use the real-time VLM microservice on stored or streamed video for captions, alerts, streams, or OpenAI-compatible completions.
- Query video analytics data and metrics from Elasticsearch via the VA-MCP server.
- Search video archives using natural language, multi-embedding fusion, and VLM critique.
- Summarize a video through chunking, dense captioning, and aggregation using the Long Video Summarization (LVS) microservice.
How the search engine actually works
The headline feature of VSS 3 is its fusion search architecture, which is a significant step beyond single-vector retrieval. Traditional video search converts a query into one embedding and finds the nearest matches in a single index. That breaks down the moment a query has multiple attributes , say, "a worker in a yellow vest climbing a ladder near a forklift."
The updated search agent workflow introduces attribute search, multi-embedding fusion search, and a critic agent to review search results. In practice, this means the system decomposes a complex query into sub-queries, runs each against separate embedding indexes, fuses the ranked results, and then runs a VLM-based critic pass to verify the clips before returning them to the user.
The real-time processing layer that feeds all of this is built on three microservices:
- RT-CV: leverages NVIDIA DeepStream SDK with models like RT-DETR, Grounding DINO, and Sparse4D to perform real-time object detection, classification, and multi-object tracking on single or multi-camera streams.
- RT-Embedding: generates semantic embeddings from video, images, and live RTSP streams using Cosmos-Embed1 models, enabling efficient video search and similarity matching.
- RT-VLM: applies Vision Language Models such as Cosmos Reason1/2 and Qwen3-VL to generate natural language captions, detect incidents, and identify anomalies in video streams.
Five workflow profiles, one modular stack
VSS is designed around a Docker Compose-based modular developer profile system: a base agent deploys in under five minutes, and additional workflows are layered on top as needed. The five profiles map cleanly to different operational needs:
| Profile | What it does |
|---|---|
| base | VLM-based Q&A and report generation on short clips |
| alerts (verification) | CV pipeline + behavior analytics + VLM verification to reduce false positives |
| alerts (VLM) | Continuous VLM anomaly detection on live streams |
| search | Agentic multi-embedding search across video archives |
| lvs | Chunked summarization of extended recordings |
Real numbers on real hardware
The performance benchmarks are worth paying attention to. For the agentic search workflow on a single GPU, an H100 handles 33 concurrent streams with a retrieval latency of 2.24 seconds, while an RTX PRO 6000 actually edges it out at 51 concurrent streams and 1.87 seconds retrieval latency. For alert verification, an H100 scales to 147 concurrent streams with sub-second verification latency.
On the summarization side, users can summarize a video 100x faster than watching in real time , for example, an hour-long video can be summarized in text in less than one minute.
A concrete example: warehouse safety compliance
The demo scenario in the official documentation is a good illustration of what this actually unlocks. A developer points OpenClaw at three 10-minute warehouse videos and asks, in plain English, to find every instance of a worker climbing a ladder and verify they are wearing a hardhat and safety vest. OpenClaw uses the VSS skills to upload the video files to VIOS, ingest the videos through the embedding microservices to generate searchable indexes, and then use the fusion search capability to find the video clips where a worker wearing a hardhat and safety vest is climbing a ladder. The agent returns a concise report with screenshots , no custom code written.
This kind of workflow has obvious reach across industries. Global systems integrators and technology solutions providers including Accenture, Dell Technologies, and Lenovo are bringing the NVIDIA Blueprint for visual search and summarization to businesses and cities worldwide, targeting factories, warehouses, shops, airports, traffic intersections, and more.
Getting started
Developers can automate VSS deployment and video analysis using coding agents like Codex and OpenClaw, which simplify setup and interaction through chat interfaces. The skills are available now on the VSS GitHub repository, and the fastest path to a running system is the NVIDIA Brev Launchable, which handles cloud provisioning. You will need an NGC API key and a compatible GPU (H100, RTX PRO 6000, AGX Thor, or DGX Spark are all supported). NVIDIA NemoClaw, an open-source stack that adds privacy and security controls to OpenClaw, can also be used to deploy video agents.
The broader implication here is architectural. The Metropolis Blueprint for VSS, NVIDIA TAO, and Video Augmentation skills help extract insights from massive volumes of video data, fine-tune models, and automate the build-and-evaluate loop , giving researchers a more repeatable way to develop reasoning vision AI agents that can detect events, reason over complex scenes, summarize activity, and send alerts. The shift from manually wired microservices to agent-driven deployment is the same pattern playing out across the software stack, and video intelligence is now fully inside that wave.