Hugging Face Ships an Open-Source Voice Agent That Ditches OpenAI's Realtime API
Hugging Face's speech-to-speech library gives you a fully local, OpenAI Realtime-compatible voice agent pipeline where every component is swappable and free
PRO- Hugging Face's speech-to-speech library is a modular VAD→STT→LLM→TTS pipeline that exposes an OpenAI Realtime-compatible WebSocket API.
- Every component is swappable: choose from Parakeet TDT, Whisper, Kokoro, Qwen3-TTS, llama.cpp, vLLM, or any OpenAI-compatible LLM endpoint.
- The pipeline runs in production as the voice backend for thousands of Reachy Mini robots, with no cloud dependency required.
- Existing OpenAI Realtime clients can point to the local server with a one-line URL change — no client code rewrite needed.
- Installs via
pip install speech-to-speech; fully local setup uses Gemma 4 via llama.cpp for the LLM and Qwen3-TTS for audio output. - Best fit for privacy-first, high-volume, or embedded deployments; native multimodal APIs (OpenAI, Gemini) still have lower raw latency for cloud-connected use cases.
Building a voice agent today usually means one of two things: hand over your audio to OpenAI's Realtime API and pay per token, or stitch together a fragile DIY pipeline of mismatched libraries. Hugging Face's speech-to-speech library is a third option , a production-grade, fully modular voice pipeline that runs entirely on your own hardware and speaks the same WebSocket protocol as OpenAI's Realtime API.
The pipeline is a cascade of four components , VAD, STT, LLM, and TTS , exposed through an OpenAI Realtime-compatible WebSocket API. Every component is swappable, and the LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at HF Inference Providers, or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack. The whole thing installs with a single pip command and starts serving on ws://localhost:8765/v1/realtime.
It's already running in production
This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots. That's not a toy benchmark , it's a real-world stress test across a fleet of physical devices, each needing low-latency, interruptible, multilingual voice interaction. The stack is powered by the cascaded VAD → STT → LLM → TTS pipeline that exposes a Realtime API-compatible /v1/realtime WebSocket.
The setup enables expressive conversational behaviors , head movements, antenna wiggles, interruptible low-latency responses , with no cloud dependency. The fact that Hugging Face is dogfooding this in hardware products gives it a credibility that most research repos lack.
The four-stage pipeline, explained
Each stage runs in its own thread and communicates via queues, keeping latency low and components independent. Here's what runs by default:
- VAD (Voice Activity Detection): Silero VAD v5 detects when you start and stop speaking, handling turn-taking and interruption.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
