ElevenLabs' Speech Engine Turns Any Chat Agent Into a Voice Agent Instantly

Building a voice agent today usually means one of two things: either you hand everything over to a fully managed platform and lose control of your LLM, or you spend weeks stitching together speech-to-text, text-to-speech, turn detection, and interruption handling yourself. ElevenLabs just released a third option. Speech Engine is a new product that bolts a complete voice layer onto any existing chat agent, and the new Speech Engine Skill makes the setup a single terminal command.

One command to go from chat to voice

The ElevenLabs Skills repo is a collection of agent skills, which are structured folders of instructions and scripts that AI coding assistants like Cursor or Claude Code can load and execute. Installing the Speech Engine Skill looks like this:

npx skills add elevenlabs/skills

The skill sets up everything you need so you can go from chat to voice in a single prompt. The underlying Agent Skills format, defined at agentskills.io, works by progressively loading context into the coding assistant: only the skill name and description are loaded at startup, with full instructions pulled in only when the skill is actually needed. This keeps context usage minimal while still giving the LLM everything it needs to scaffold your integration correctly.

What Speech Engine actually does

ElevenLabs Speech Engine adds voice capabilities to any chat agent. ElevenLabs handles speech-to-text and text-to-speech while your server provides the LLM logic. The connection model is straightforward:

1. A user speaks in the browser. ElevenLabs captures the audio and transcribes it. The transcript is sent to your server along with the full conversation history.
2. Your server passes the transcript to your LLM and streams the response back.
3. ElevenLabs converts the text to speech and plays it in the browser.

The SDK manages connection lifecycle, turn-taking, and interruption detection so you can focus on your agent's behavior. All of this runs over a WebSocket, where each connection maps to one conversation session.

The pipeline under the hood

Speech Engine is not a thin wrapper. It combines ElevenLabs' leading speech, transcription, and voice orchestration models into a single pipeline, all custom built to work best together. The individual components are:

• Speech-to-text (Scribe v2): Conversational transcription optimized for accuracy at ultra-low latency, supporting 90+ languages.
• Turn detection: Uses real-time signals from Scribe v2 Realtime, including emotional cues and speech patterns, to determine when an agent should speak, pause, or wait.
• Interruption handling: When the user speaks mid-response, the SDK cancels the in-flight LLM request automatically via an AbortSignal (TypeScript) or task cancellation (Python).
• Text-to-speech (Eleven v3 Conversational): An ultra-low-latency version of Eleven v3, optimized for live, back-and-forth dialogue.
• Voice activity detection: Filters speech from background noise at the input level, so only clean audio reaches the transcription model.

A minimal server integration in TypeScript looks like this:

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import "dotenv/config";
const elevenlabs = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY,
});
// Attach Speech Engine to your WebSocket endpoint
const engine = await elevenlabs.speechEngine.create({
  name: "My Speech Engine",
  speechEngine: {
    wsUrl: "wss://your-server.example.com/ws",
  },
});
// Your server's onTranscript callback:
// engine.on("transcript", async ({ transcript, session }) => {
//   const stream = await openai.chat.completions.create({ ... });
//   session.sendResponse(stream); // SDK extracts text automatically
// });

Any LLM, no rearchitecting

Your agent's LLM, RAG, and architecture all remain untouched. You can use OpenAI, Anthropic, Google Gemini, or any model that produces text. The SDK auto-extracts text from OpenAI, Anthropic, and Gemini stream formats. For anything else, you can pass a plain string or an async iterable of chunks.

On the server side, in TypeScript you can attach Speech Engine to any Node.js HTTP server (Express, Fastify, or plain http.createServer()), or run a standalone WebSocket server. Python developers can use engine.serve() for a standalone server, or integrate with FastAPI or any ASGI framework via engine.create_session().

Speech Engine vs. ElevenAgents

ElevenLabs already has a fully managed voice agent platform called ElevenAgents. The distinction is deliberate. ElevenAgents is a fully hosted platform where ElevenLabs provides the LLM, knowledge base, and tools. Speech Engine is for developers who want to bring their own LLM and control the conversation logic on their own server. The client SDK is intentionally shared between both, so if you start with Speech Engine and later want to migrate to the fully managed path, no client-side changes are needed.

What it's good for

Speech Engine is designed for developers who want to bring their own LLM and control the conversation logic on their own server. Concrete use cases that have already emerged from ElevenLabs' own hackathon include:

• A voice trading assistant that connects a custom LLM to a live portfolio via Alpaca
• An interactive voice CV where recruiters talk to an AI version of a candidate instead of reading a resume
• A 24/7 AI radio station that takes real calls and maintains per-caller memory across sessions
• A public speaking coach with six distinct AI hecklers that respond contextually to what you actually say

The pattern in all of these is the same: a developer already had domain-specific LLM logic they wanted to keep, and Speech Engine gave them voice without touching it.

The real shift here

The harder problem in voice agents has never been TTS quality. It has been the plumbing: detecting when a user is done speaking versus just pausing, handling mid-sentence interruptions without leaving the LLM hanging, and keeping audio latency low enough that the conversation feels natural. ElevenLabs' voice models are optimized for conversation, delivering ultra-low latency in real-world environments, and dedicated models handle overlapping speech and mid-sentence changes without custom logic on your end.

The Skills packaging is also worth noting separately. Rather than shipping a library you integrate manually, ElevenLabs is distributing this as a coding-assistant-native artifact. By default, only the name and description from the frontmatter are loaded into the context window, taking up very little context, but giving the LLM information about what type of skills are available. When a skill is decided to be useful, the full SKILL.md is loaded, and then only when necessary are the other files in the scripts, references, and assets folders loaded into context. It's a small but meaningful bet on the idea that the IDE is becoming the primary integration surface for developer tooling.

Speech Engine is available now. Eleven v3 Conversational is priced at $0.08 per minute , in line with other ElevenLabs TTS models in the Agents tier. The platform supports SOC 2, HIPAA, and GDPR compliance, with EU Data Residency and Zero Retention Mode available for stricter data control.