Skip to main content
Back

Tencent's Youtu-Parsing-Omni Turns Seven Media Types Into One JSON Output

Tencent's 5B omni-modal parser handles documents, charts, geometry, audio and video into one structured JSON schema, rivaling Gemini-3-Pro.

  • Tencent released Youtu-Parsing-Omni, a 5B omni-modal parser for documents, images, audio and video.
  • Single JSON schema (OmniSchema) covers seven input types including charts, geometry, flowcharts and text-rich video.
  • 96.96 Overall on OmniDocBench v1.6, state-of-the-art, beating Gemini-3-Pro (92.91) and GPT-5.2 (86.59).
  • 75.08 average on OmniParsingBench, second only to Gemini-3-Pro among all evaluated systems.
  • Novel OSAD post-training: self-distillation splits tokens into content vs structure with different losses.
  • Open weights with vLLM plugin, OpenAI-compatible serving, custom Youtu license.

Tencent’s Youtu Lab has released Youtu-Parsing-Omni, a 5-billion-parameter model that parses documents, natural images, charts, flowcharts, geometry figures, audio, and audiovisual video. It returns each input through OmniSchema, a unified JSON format designed for structured extraction.

Multimodal extraction pipelines commonly combine optical character recognition, layout analysis, chart conversion, speech recognition, and video captioning systems. Youtu-Parsing-Omni consolidates those functions behind one model and output contract. Tencent has published the weights, inference code, architecture details, and benchmark results under a custom Youtu license.

Pixels and sound share an encoder

Youtu-Parsing-Omni replaces separate visual and audio towers with a shared Youtu-Omni-Encoder initialized from a pretrained text language model. That initialization is intended to preserve strong text recognition while extending the encoder to visual and acoustic inputs.

Comparison of a two-tower multimodal parser and the shared Youtu-Parsing-Omni encoder
Youtu-Parsing-Omni shares most perception parameters across visual and audio inputs.

Small modality-specific stems convert image patches and log-mel audio frames into tokens. A shared bidirectional Transformer then processes packed combinations of images, audio chunks, and video using a common (t, h, w) positional system. Modality-specific mergers and projectors pass the resulting tokens, along with the prompt, to the Youtu-LLM decoder, which generates OmniSchema JSON.

Video synchronization begins inside the encoder. Non-fusion layers give each frame and audio chunk an independent attention window, while selected fusion layers open windows at audio-to-frame boundaries. Visual tokens use the frame index for t and patch coordinates for