Alibaba's Fun-Realtime-AudioChat Beats OpenAI on Voice Reasoning and Conversation

Alibaba's Tongyi Lab just landed three simultaneous top rankings on the Artificial Analysis speech-to-speech leaderboard, the most comprehensive independent benchmark for real-time voice models. Their Fun-Realtime-AudioChat model took first place in both speech reasoning (97.6% on Big Bench Audio) and conversational dynamics (97.8% on Full Duplex Bench), while Fun-Realtime-ASR claimed the top spot in transcription accuracy with an AA-WER (word error rate index) of just 1.8%. These aren't narrow wins either: GPT-Realtime-2 (High) scores 96.6% on reasoning and 95.3% on conversational dynamics, meaning Fun-Realtime-AudioChat beats OpenAI's best on both axes simultaneously.

Two models, one story

The Fun series is really two distinct products solving different problems. Fun-Audio-Chat is an open-source end-to-end speech-to-speech model developed by Alibaba's Tongyi Bailing team that can understand and respond to voice input directly, without needing separate ASR, LLM, and TTS components. Fun-ASR, on the other hand, is a dedicated transcription engine. Fun-ASR v1.5 is a 30B-parameter MoE-based end-to-end speech recognition model trained on tens of millions of hours of real speech data, systematically advancing language coverage, dialect recognition depth, and text output quality.

The distinction matters for builders. Fun-Audio-Chat is the conversational layer: you talk to it, it talks back, and it understands your emotion. Fun-ASR is the transcription workhorse: you pipe audio in, you get accurate, well-formatted text out. Both are now available via Alibaba Cloud's Model Studio API.

The architecture that makes it tick

Existing joint speech-text models face critical challenges: the temporal resolution mismatch between speech tokens (typically 25Hz) and text tokens (approximately 3Hz) dilutes semantic information and hinders the full utilization of the LLM's core capabilities, while continual pre-training often leads to catastrophic forgetting of the text LLM's knowledge. Fun-Audio-Chat was built specifically to solve these two problems.

The core innovation is called Dual-Resolution Speech Representations (DRSR). The Shared LLM processes audio at an efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Think of it as a two-tier pipeline: the backbone quickly grasps meaning at low resolution, and a specialized head adds the fine-grained acoustic detail needed for natural-sounding output.

The second innovation is Core-Cocktail Training. This method preserves the underlying text LLM's capabilities during multimodal training. Previous models suffered from "catastrophic interference" where adding speech capabilities degraded text performance; Core-Cocktail prevents this by addressing temporal resolution mismatches between modalities. The practical result: the model reasons well in audio because it hasn't forgotten how to reason in text.

Here's how you'd run inference on the open-source 8B model:

from transformers import AutoModelForCausalLM, AutoProcessor
import torch
processor = AutoProcessor.from_pretrained("FunAudioLLM/Fun-Audio-Chat-8B")
model = AutoModelForCausalLM.from_pretrained(
    "FunAudioLLM/Fun-Audio-Chat-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
# Pass audio directly , no ASR step needed
inputs = processor(audio=your_audio_array, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
response_audio = processor.decode(outputs[0])

What "full duplex" actually means here

Full duplex means the model can listen and speak at the same time, just like a human. Most voice AI systems are half-duplex: they wait for you to stop talking before they respond. The parallel speech-text input stream architecture allows the model to accept user speech when the assistant is generating speech, effectively utilizing the time slots that would otherwise be idle, and is designed to handle both user and assistant speech inputs simultaneously.

The Conversational Dynamics benchmark (Full Duplex Bench) specifically tests four behaviors that separate good voice AI from bad:

• Pause handling: not interrupting when the user pauses mid-sentence
• Turn-taking: correctly picking up the conversation when the user finishes
• Interruption handling: stopping and addressing the user when they cut in
• Backchannel handling: continuing to speak when the user says "yeah" or "mm-hmm" rather than treating it as a turn handoff

Fun-Realtime-Audiochat leads with a conversational dynamics score of 97.8% on the Full Duplex Bench dataset across 19 models evaluated, with GPT-Realtime-2 (Minimal) following at 96.1% and GPT-Realtime-1.5 at 95.7%.

Fun-ASR: the transcription engine underneath

The ASR side of the story is equally impressive. Fun-ASR v1.5 supports high-precision recognition of 30 languages, fully covers seven major Chinese dialects and over 20 regional accents, and introduces dedicated optimization for classical Chinese poetry recitation, while significantly improving punctuation prediction and text normalization (ITN).

The dialect coverage is unusually deep. Fun-ASR v1.5 comprehensively supports seven major Chinese dialect systems , Wu, Cantonese, Min, Hakka, Gan, Xiang, and Jin , and covers over 20 regional accents, achieving a 56.2% relative reduction in CER compared to the previous version, trained on over 500,000 hours of real dialect speech data.

The model also handles what most ASR systems fail on: text normalization (ITN), which converts spoken numbers and dates into the formats humans expect in written text. For example, the spoken phrase "er ling er liu nian san yue er shi jiu hao" becomes "2026年3月29日" automatically. These improvements significantly reduce post-editing costs, particularly suitable for meeting minutes generation, news interview transcription, and legal records where text standardization is required.

What it's good at , and where it's still rough

The benchmarks paint a clear picture of strengths:

• Reasoning over audio: 97.6% on Big Bench Audio, a dataset of 1,000 spoken logic and reasoning questions
• Conversational dynamics: #1 across all 19 models tested for natural turn-taking and interruption handling
• Transcription accuracy: 1.8% WER, the lowest on the leaderboard
• Efficiency: The model achieves competitive performance in spoken question answering, audio understanding, and full-duplex interaction, while reducing training costs by approximately 50%.
• Voice function calling: SOTA results on Speech-ACEBench, Speech-BFCL, and Speech-SmartInteract benchmarks.

The limitations are real and the team is transparent about them. The full-duplex mode is still under active development and the demo page explicitly warns that the model may generate hallucinations or produce inconsistent responses in complex or ambiguous scenarios. The realtime AudioChat variant is also not yet publicly available via API , the tweet notes it is "coming soon."

The bigger shift this represents

The traditional approach to voice AI has been a "cascade" system involving three distinct steps: ASR converts voice to text, an LLM processes that text to generate a response, and TTS synthesizes that response back into audio. While effective for basic commands, this results in high latency and the loss of paralinguistic information , the sighs, the tone, the urgency, and the emotion that carry half the meaning of human speech.

Fun-Audio-Chat's end-to-end approach skips that pipeline entirely. The model demonstrates strong empathy capabilities, understanding emotional context in voice interactions and responding with appropriate empathetic expressions without explicit emotional markers, and can naturally detect and respond to user emotions through paralinguistic cues such as tone, pace, and prosody. That's a capability that is structurally impossible in a cascade system, because the ASR step discards all acoustic information before the LLM ever sees it.

The competitive context is also notable. The top speech-to-speech models by reasoning quality are: Fun-Realtime-Audiochat (97.6%), Step-Audio R1.1 Realtime (97.6%), Grok Voice Think Fast 1.0 (97.1%), GPT-Realtime-2 High (96.6%), and Gemini 3.1 Flash Live Preview High (96.6%). Alibaba is now sitting at the top of a leaderboard that includes OpenAI, Google, and xAI , and doing it with an open-source model that runs on a single 24GB GPU for inference.

How to get started

There are two paths depending on what you need:

1. Open-source self-hosted: The Fun-Audio-Chat-8B weights are on HuggingFace and ModelScope under Apache 2.0. The architecture requires approximately 24GB GPU memory for inference and 4×80GB for training, making it accessible to researchers and mid-sized companies without requiring massive infrastructure.
2. Managed API: Fun-ASR is available now via Alibaba Cloud Model Studio with WebSocket streaming support. Available model variants include fun-asr-realtime (stable), fun-asr-realtime-2026-02-28 (latest snapshot), and fun-asr-flash-8k-realtime for telephony-grade 8kHz audio. Fun-Realtime-AudioChat API access is listed as coming soon.

The open-source 8B model also supports LoRA fine-tuning on 4×80GB GPUs, so teams can adapt it to domain-specific vocabulary or custom personas without starting from scratch. For voice agent builders, the combination of #1 transcription accuracy, full-duplex conversation, and native function calling from voice is a meaningful stack upgrade , especially at a price point that starts at free.