Skip to main content
Back

HKU's VideoRAG Turns Hundreds of Video Hours Into a Queryable Knowledge Base

VideoRAG brings RAG to video at scale, with a desktop app that can query hundreds of hours of footage using a knowledge graph architecture

Pro

  • VideoRAG is an open-source RAG framework for querying video libraries of any length using natural language, accepted at KDD 2026.
  • A dual-channel architecture combines a knowledge graph over transcripts/entities and multi-modal visual embeddings for retrieval.
  • Runs on a single RTX 3090 (24GB) and can process hundreds of hours of video; a 1-hour video indexes in ~40-50 minutes.
  • Vimo, the companion desktop app (Electron + Python), is in beta for macOS Apple Silicon, with Windows/Linux coming soon.
  • Outperforms MiniCPM-o and VideoAgent on the new LongerVideos benchmark (134+ hours, 164 videos, 602 queries).
  • Fully open-source at github.com/HKUDS/VideoRAG; supports GPT-4, Claude, and local models via Ollama.

VideoRAG is an open-source framework from the Hong Kong University Data Science Lab that lets you query video libraries using natural language, no matter how long the videos are. The project now ships with Vimo, a cross-platform desktop app that wraps the research into a drag-and-drop interface. The combination is significant: it is one of the first systems to treat a collection of long videos as a structured, queryable knowledge base rather than a pile of files.

Developed by HKUDS and accepted for publication at KDD 2026, one of data mining's most prestigious conferences, this is not a weekend demo. RAG has demonstrated remarkable success in enhancing LLMs through external knowledge integration, yet its application has primarily focused on textual content, leaving the rich domain of multi-modal video knowledge predominantly unexplored. VideoRAG is the direct answer to that gap.

The wall that existing models hit

Traditional large video-language models struggle with long durations due to context limits and computational overhead, often losing semantic continuity across scenes or episodes. The naive workaround is to sample frames at fixed intervals, but that throws away most of the signal. Current video understanding approaches treat long videos as isolated clips rather than as interconnected knowledge networks. The result is a model that can answer questions about a single scene but falls apart when you ask it to synthesize information across a six-hour lecture series.

Traditional approaches break down with long videos. Frame-by-frame analysis is computationally prohibitive. Scene detection lacks semantic understanding. Keyword-based metadata is tedious to create and misses visual context.