---
title: "ByteDance's video-SALMONN 2 Beats GPT-4o by Teaching AI to Hear Videos"
subtitle: "ByteDance and Tsinghua open-source video-SALMONN 2, a 7B audio-visual model that beats GPT-4o and Gemini-1.5 Pro at video captioning using a novel multi-round DPO training technique"
url: "https://alphasignal.ai/news/bytedance-s-video-salmonn-2-beats-gpt-4o-by-teaching-ai-to-hear-videos"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-07-13T08:38:48.000Z"
updated: "2026-07-13T19:52:00.505Z"
type: "Repo"
topics:
  - "Audio"
  - "Llms"
subtopics:
  - "Realtime Voice"
  - "Speech To Text"
  - "Vision Language"
upvotes: 2852
source_url: "https://github.com/bytedance/SALMONN"
comments: 0
truncated: true
---

# ByteDance's video-SALMONN 2 Beats GPT-4o by Teaching AI to Hear Videos

> ByteDance and Tsinghua open-source video-SALMONN 2, a 7B audio-visual model that beats GPT-4o and Gemini-1.5 Pro at video captioning using a novel multi-round DPO training technique

Most video models are effectively deaf. They sample frames, ignore the audio track, and call it video understanding. [video-SALMONN 2](https://github.com/bytedance/video-SALMONN-2), the latest release from ByteDance and Tsinghua University's joint research team, takes a different bet: that truly understanding a video means seeing **and** hearing it simultaneously. The result is a family of open-source audio-visual models that, at 7B parameters, outperforms GPT-4o and Gemini-1.5 Pro on video captioning tasks.

## The problem no one was measuring

Generating accurate and comprehensive video captions remains a significant challenge. Videos encode complex content spanning both spatial features within individual frames and audio-visual events across temporal sequences, and despite the prevalence of incompleteness and hallucinations in video captions, few studies have addressed caption quality, largely due to the lack of reliable quantitative evaluation metrics. On top of that, although audio is typically paired with video and provides vital complementary information, most existing visual LLMs lack audio processing capabilities, further contributing to deficiencies in existing video captioning systems.

The team's diagnosis is sharp: if your training data has bad captions, your model learns bad understanding. Fixing captioning quality is therefore not just a downstream task but a lever for improving general video comprehension.

## What's actually in the box

video-SALMONN 2 is a powerful audio-visual large language model that generates high-quality audio-visual video captions, developed by the Department of Electronic Engineering at Tsinghua University and ByteDance. The model family ships in three sizes:

* **3B** , built on Qwen 2.5-VL 3B
* **7B** , the primary model, built on LLaVA-OneVision-7B, processes video at 1 FPS up to 110 frames
* **72B** , built on Qwen 2.5-VL 72B, the open-source state-of-the-art

The architecture grafts a separate audio encoder branch onto a pre-trained visual LLM. The 3B and 7B models achieve SOTA results at comparable scales, while the 72B model surpasses all other open-source systems. Specifically, the largest video-SALMONN 2+ (72B) consistently performs better than GPT-4o and Qwen3-Omni-Flash, while surpassing Gemini-1.5 Pro on most audio-visual QA benchmarks.

## Seeing and hearing at the same time

The architecture is straightforward in concept but careful in execution. Video frames are processed by a visual encoder, while the audio waveform runs through a separate Whisper-Large-v3 encoder. Both streams are projected into the LLM's input space and then

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [bytedance/SALMONN — GitHub](https://github.com/bytedance/SALMONN)
- [bytedance/video-SALMONN-2 — GitHub](https://github.com/bytedance/video-SALMONN-2)
- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models — arXiv](https://arxiv.org/html/2506.15220v3)
- [video-SALMONN 2 — HuggingFace Paper Page](https://huggingface.co/papers/2506.15220)
- [video-SALMONN 2 — OpenReview (ICLR 2026 submission)](https://openreview.net/forum?id=O2tca5tExP)