Kyutai's MoshiRAG Fixes Voice AI Hallucinations While Still Talking
Kyutai's new system lets Moshi trigger web search or LLM lookups mid-sentence, closing the factuality gap with slower turn-based voice assistants.
PRO- MoshiRAG is the first full-duplex speech model with retrieval-augmented generation, built on Kyutai's Moshi.
- Model emits a special
<ret>token to trigger asynchronous retrieval while continuing to speak filler content. - Retrieval completes in under 2 seconds, hidden by the natural gap before the keyword in a response.
- Boosts TriviaQA accuracy from 22.8% (vanilla Moshi) to 73.7%, matching most non-duplex speech LMs.
- Back end is swappable at inference: local Gemma, GPT-4.1, or Tavily web search, no retraining needed.
- Generalizes to unseen math tasks, reaching 43.2% on GSM8K by using an LLM as an external tool.
Full-duplex voice models can listen and talk at the same time, which makes them feel more human than the turn-based pipelines that dominate today's voice assistants. The catch is that they hallucinate. Kyutai's new MoshiRAG tackles that problem by giving Moshi the ability to fire off a retrieval call while it's already talking, then quietly stitch the retrieved facts into the rest of its answer before anyone notices.
The system matches or beats most publicly released non-duplex speech language models on question answering, while keeping the low latency and interruption handling of a duplex architecture. Code and demos are out, and the design is deliberately modular so you can swap the knowledge source without retraining.
Why speech models flunk trivia night
Speech-to-speech models train on far less text-equivalent data than text LLMs, so they lag on anything that needs concrete facts. Recent studies show native audio models struggle more than text models on factuality tasks like question answering, largely because of the smaller volumes of speech data available for pretraining. Scaling up would help, but a bigger model breaks real-time inference on a single GPU.
Retrieval-augmented generation is the obvious fix for text models. Timing is the awkward part. A full-duplex model produces audio continuously at 12.5 Hz, and any pause to wait on a retrieval call shows up as an unnatural silence. Concurrent work such as StreamRAG and KAME also exploits temporal gaps in spoken conversations to perform retrieval, but StreamRAG is restricted to non-full-duplex settings and KAME relies on frequent, fixed-interval LLM calls that ignore conversational context.
Hiding retrieval behind "let me think"
Humans don't blurt out the answer immediately. They say something like "good question, let me think" before the actual keyword arrives. MoshiRAG exploits that natural temporal gap between the onset of a spoken response and the emergence of its key informational content, which the authors call the keyword delay. If retrieval finishes before the model reaches the meaningful part of its reply, the user hears one continuous, grounded response.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
