Moonshot AI's Kimi-Audio Beats GPT-4o on Emotion Control Trained on 13M Hours
Moonshot AI open-sources Kimi-Audio-7B-Instruct, a single model that handles speech recognition, emotion detection, audio Q&A, and live voice chat — beating GPT-4o-mini on conversation quality.
PRO- Moonshot AI releases Kimi-Audio-7B-Instruct, an open-source model handling ASR, emotion recognition, audio Q&A, captioning, and live voice chat in one checkpoint.
- Pre-trained on 13 million hours of speech, music, and environmental audio — roughly 19x the scale of Whisper large-v3.
- Scores 3.90/5 on human speech conversation evals, beating GPT-4o-mini (3.45) and GLM-4-Voice (3.65); only GPT-4o (4.06) scores higher.
- Sets new SOTA on LibriSpeech (1.28 WER), AISHELL-1 (0.60 WER), and multiple audio understanding benchmarks including MMAU and MELD.
- Requires ~24 GB VRAM at full precision or ~8 GB with 4-bit quantization; English and Mandarin only for speech generation.
- MIT-licensed with training code, weights, and a standardized evaluation toolkit all released open-source.
Kimi-Audio-7B-Instruct is Moonshot AI's open-source answer to a long-standing fragmentation problem in audio AI: you've always needed separate models for transcription, emotion analysis, sound classification, and voice chat. Kimi-Audio collapses all of that into one 7B-parameter checkpoint, trained from scratch on a scale that most audio labs can't match.
One model, every audio task
Kimi-Audio is designed as a universal audio foundation model capable of handling a wide variety of audio processing tasks within a single unified framework, covering speech recognition (ASR), audio question answering (AQA), audio captioning (AAC), speech emotion recognition (SER), sound event and scene classification, and end-to-end speech conversation. That's not a feature list padded with stretch goals , the model ships a single set of weights that handles all of them.
The practical upshot: you load one model and route different audio tasks through it using natural-language instructions. No pipeline stitching, no separate ASR + emotion + TTS services to maintain.
The architecture: two representations are better than one
The core insight behind Kimi-Audio's design is that audio carries two kinds of information simultaneously , what is being said (semantics) and how it sounds (acoustics). Most prior models picked one or the other. Kimi-Audio uses both at once.
It adopts a hybrid input representation combining Whisper-derived continuous acoustic features and discrete semantic tokens at 12.5Hz, ensuring rich perception and efficient modeling. This integrated architecture enables Kimi-Audio to seamlessly handle diverse audio-language tasks from speech recognition and understanding to speech conversation within a single unified model framework. The tokenization allows the model to leverage the efficiency and semantic focus of discrete tokens while benefiting from the rich acoustic details captured by continuous representations.
The three-part system works like this:
- Audio Tokenizer: Converts raw audio into discrete semantic tokens via vector quantization (at 12.5Hz), then extracts continuous acoustic vectors from a Whisper encoder and adds them on top. The combined signal is what the LLM actually sees.
- Audio LLM: A transformer-based model initialized from a pre-trained text LLM (Qwen 2.5-7B) with shared layers processing multimodal inputs, followed by parallel heads for autoregressively generating text tokens and discrete audio semantic tokens.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
