ByteDance's video-SALMONN 2 Beats GPT-4o by Teaching AI to Hear Videos
ByteDance and Tsinghua open-source video-SALMONN 2, a 7B audio-visual model that beats GPT-4o and Gemini-1.5 Pro at video captioning using a novel multi-round DPO training technique
PRO- video-SALMONN 2 open-sourced: ByteDance + Tsinghua release audio-visual LLMs in 3B, 7B, and 72B sizes under Apache 2.0. GitHub
- Beats GPT-4o and Gemini-1.5 Pro: The 7B model achieves lower captioning error rates than both proprietary systems; 72B leads all open-source models on audio-visual QA benchmarks.
- Core innovation is MrDPO: Multi-round Direct Preference Optimisation with periodic LoRA proxy re-initialisation reduces captioning error rates by 28% vs single-round DPO.
- Audio-first architecture: Whisper-Large-v3 audio encoder is interleaved with visual tokens per second, enabling true joint audio-visual understanding rather than audio as an afterthought.
- Data flywheel effect: MrDPO-trained captioner re-annotates 100k training videos; models trained on this data (video-SALMONN 2+) outperform models 2-10x their size.
- Fully reproducible: Checkpoints, training scripts, and a 483-video human-annotated benchmark released on HuggingFace; atomic-event evaluator works with local Qwen3-4B.
Most video models are effectively deaf. They sample frames, ignore the audio track, and call it video understanding. video-SALMONN 2, the latest release from ByteDance and Tsinghua University's joint research team, takes a different bet: that truly understanding a video means seeing and hearing it simultaneously. The result is a family of open-source audio-visual models that, at 7B parameters, outperforms GPT-4o and Gemini-1.5 Pro on video captioning tasks.
The problem no one was measuring
Generating accurate and comprehensive video captions remains a significant challenge. Videos encode complex content spanning both spatial features within individual frames and audio-visual events across temporal sequences, and despite the prevalence of incompleteness and hallucinations in video captions, few studies have addressed caption quality, largely due to the lack of reliable quantitative evaluation metrics. On top of that, although audio is typically paired with video and provides vital complementary information, most existing visual LLMs lack audio processing capabilities, further contributing to deficiencies in existing video captioning systems.
The team's diagnosis is sharp: if your training data has bad captions, your model learns bad understanding. Fixing captioning quality is therefore not just a downstream task but a lever for improving general video comprehension.
What's actually in the box
video-SALMONN 2 is a powerful audio-visual large language model that generates high-quality audio-visual video captions, developed by the Department of Electronic Engineering at Tsinghua University and ByteDance. The model family ships in three sizes:
- 3B , built on Qwen 2.5-VL 3B
- 7B , the primary model, built on LLaVA-OneVision-7B, processes video at 1 FPS up to 110 frames
- 72B , built on Qwen 2.5-VL 72B, the open-source state-of-the-art
The architecture grafts a separate audio encoder branch onto a pre-trained visual LLM. The 3B and 7B models achieve SOTA results at comparable scales, while the 72B model surpasses all other open-source systems. Specifically, the largest video-SALMONN 2+ (72B) consistently performs better than GPT-4o and Qwen3-Omni-Flash, while surpassing Gemini-1.5 Pro on most audio-visual QA benchmarks.
Seeing and hearing at the same time
The architecture is straightforward in concept but careful in execution. Video frames are processed by a visual encoder, while the audio waveform runs through a separate Whisper-Large-v3 encoder. Both streams are projected into the LLM's input space and then
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
