Tencent’s 5B parser turns seven media types into one JSON schema

Tencent’s Youtu Lab has released Youtu-Parsing-Omni, a 5-billion-parameter model that parses documents, natural images, charts, flowcharts, geometry figures, audio, and audiovisual video. It returns each input through OmniSchema, a unified JSON format designed for structured extraction.

Multimodal extraction pipelines commonly combine optical character recognition, layout analysis, chart conversion, speech recognition, and video captioning systems. Youtu-Parsing-Omni consolidates those functions behind one model and output contract. Tencent has published the weights, inference code, architecture details, and benchmark results under a custom Youtu license.

Pixels and sound share an encoder

Youtu-Parsing-Omni replaces separate visual and audio towers with a shared Youtu-Omni-Encoder initialized from a pretrained text language model. That initialization is intended to preserve strong text recognition while extending the encoder to visual and acoustic inputs.

Small modality-specific stems convert image patches and log-mel audio frames into tokens. A shared bidirectional Transformer then processes packed combinations of images, audio chunks, and video using a common (t, h, w) positional system. Modality-specific mergers and projectors pass the resulting tokens, along with the prompt, to the Youtu-LLM decoder, which generates OmniSchema JSON.

Video synchronization begins inside the encoder. Non-fusion layers give each frame and audio chunk an independent attention window, while selected fusion layers open windows at audio-to-frame boundaries. Visual tokens use the frame index for t and patch coordinates for (h, w); audio tokens use their time offset for