Sesame's TurnBench Exposes How Gemini Live and OpenAI Realtime Fumble Conversations

Voice AI has gotten shockingly good at sounding human, but most systems still botch the rhythm of conversation. They cut you off mid-sentence, sit awkwardly during a pause, or mistake a polite mm-hmm for an interruption. Sesame just released TurnBench, an open benchmark designed to measure exactly this skill, and its debut leaderboard shows that no current system, including Gemini Live and OpenAI Realtime, gets it right.

TurnBench is an open-source benchmark for evaluating turn-taking dynamics in spoken dialogue. It ships with a public leaderboard, an interactive conversation viewer, self-serve scoring on a dev set, a 30-hour evaluation corpus, and a 104-hour training set. The scorer and baselines live on GitHub, and all three data splits are available on Hugging Face under a non-commercial license that prohibits voice cloning.

The 200 millisecond problem

Turn-taking sounds trivial until you look at the numbers. The typical gap between turns is only ~200 milliseconds, which is shorter than the ~600 ms it takes a person to plan a single word. Humans don't react to silence, they predict it from prosody and syntax. A voice agent that waits for a hard silence will always feel slow, and one that fires too fast will trample the user.

Sesame frames the problem as a continuous binary decision: speak or stay silent. Two ambiguous moments dominate. When a user stops talking, is it a real end-of-turn or a mid-turn pause? When a user starts talking over the agent, is it a barge-in or a backchannel? Existing evaluation defaults to Switchboard, a collection of phone calls between strangers recorded in the early 90s, which does not reflect how people actually talk over dinner or during collaborative work.

What the benchmark actually scores

TurnBench grounds its labels in conversation analysis and reduces everything to four events: turns, interruptions, backchannels, and mid-turn pauses. Models are scored on two tracks:

• End-of-Turn (EOT) detection: true positives are real floor transitions, false positives are mid-turn pauses.
• Interruption (INT) detection: true positives are legitimate barge-ins, false positives are backchannels, noise, or channel bleed.

To evaluate a model's responses, the scorer searches a window around each human-labeled event (0.25s before to 3s after) and checks whether the model committed a matching event within that window. Submissions are ranked by recall, capped at a false-positive rate of 0.15 so that a trigger-happy model cannot cheat its way up the board. Latency is reported as signed milliseconds at the 10th, 50th, and 90th percentiles.

Crucially, the benchmark scores committed decisions rather than probability curves. Models that never expose probabilities, like Moshi or Gemini Live, can be evaluated on the same footing as VAD-style endpointers.

Inside the corpus

TurnBench is built on 30 hours of dual-channel dyadic English speech: 154 dialogues featuring 106 voice actors in 53 pairs. The recordings were made in a professional studio with one speaker per channel, in separate sound-isolated booths, so the two sides can be scored independently. Every dialogue is labeled by three annotators with a Fleiss' kappa of 0.78, which is strong agreement for a subjective linguistic task.

The corpus is deliberately balanced across six registers: casual talk, task-oriented dialogue, instructional exchanges, collaborative problem-solving, argumentative disagreement, and narrative storytelling. Argumentative dialogue is dense with competitive interruptions, while instructional conversations feature longer turns and fewer floor changes. Alongside the eval set, Sesame is releasing a 104-hour training set called otoSpeech, hand-labeled under the same protocol.

Nobody is winning

Sesame ran 14 systems on the benchmark, from an RMS energy VAD baseline to commercial endpointers to full-duplex dialogue models. The results are unflattering across the board.

Three findings jump out from the report. EOT recall stays consistent across conversation types, but interruption false positives spike in backchannel-heavy casual talk, which turns out to be the hardest register for current systems. There is also a real speed-accuracy tradeoff: acoustic detectors fire on the first frame of listener speech and get precision wrong, while semantic systems wait for evidence and get latency wrong. And the winner still is not good enough. Voice Activity Projection (VAP) leads both tracks (0.845 recall at 0.055 false-positive rate on end-of-turn, 0.945 at 0.107 on interruption) but takes nearly a second on average to commit an interruption.

For a human baseline, participants in the TurnBench corpus begin turn transfers a median 151 ms before the turn ends. Every model on the leaderboard is reacting after the fact.

Why this matters if you are building voice

If you ship a voice product, TurnBench gives you three things that did not exist as an integrated package before:

• A held-out test set with a labeled leaderboard so you can benchmark your endpointer against real commercial systems instead of Switchboard.
• A 104-hour training corpus labeled under the same taxonomy, which lets you train supervised turn-taking models without inventing your own annotation scheme.
• A scoring protocol that treats a deployed system as a causal decision-maker at a single operating point, matching what you actually run in production.

The limitations are honest. TurnBench is English-only, and timing norms vary wildly across languages, with average gaps ranging from around 7 ms in Japanese to 470 ms in Danish. The audio is studio-clean, so real-world noise, echo, and background chatter will degrade performance in ways this corpus cannot predict. And the interruption track scores the user's channel, which is a poor fit for full-duplex models that speak and listen simultaneously.

Sesame argues that turn-taking is now where speech synthesis was a year ago: the easy benchmarks have saturated, and the field needs a harder yardstick to tell good models from great ones. If VAP's second-long interruption latency really is the state of the art, then the next round of voice agents will be judged less by how they sound and more by whether they know when to shut up.