---
title: "AssemblyAI's Universal-3.5 Pro Gives Voice Agents Memory Between Turns"
subtitle: "AssemblyAI's Universal-3.5 Pro Realtime brings conversation memory to streaming STT, letting voice agents feed prior turns to the model mid-call without reconnecting."
url: "https://alphasignal.ai/news/assemblyai-s-universal-3-5-pro-gives-voice-agents-memory-between-turns"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "Artificial Analysis"
company_url: "https://alphasignal.ai/company/artificialanlys"
published: "2026-07-06T15:54:46.000Z"
updated: "2026-07-07T18:39:57.639Z"
type: "News"
topics:
  - "Audio"
  - "Benchmarks"
subtopics:
  - "Speech To Text"
  - "Realtime Voice"
upvotes: 115
read_time: "6 min"
source_url: "https://artificialanalysis.ai/speech-to-text/methodology"
comments: 0
---

# AssemblyAI's Universal-3.5 Pro Gives Voice Agents Memory Between Turns

> AssemblyAI's Universal-3.5 Pro Realtime brings conversation memory to streaming STT, letting voice agents feed prior turns to the model mid-call without reconnecting.

Every streaming speech-to-text model on the market today has the same blind spot: it hears audio, transcribes it, and immediately forgets everything. The next utterance arrives with zero memory of what came before. That works fine for dictation, but it quietly breaks voice agents, where each user turn is a direct reply to something the agent just said. [AssemblyAI's Universal-3.5 Pro Realtime](https://www.assemblyai.com/universal-3-pro-streaming) is the first streaming STT model to fix this at the model level, and it's available now.

## The problem no one had solved yet

Voice agent pipelines fail in predictable ways. A user says "user at gmail dot com" instead of "user@gmail.com" because the model has no idea an email was expected. A caller spells out a policy ID and the model mishears half of it. A medication name gets mangled because the model has never seen it before and has no domain context to fall back on.

A collections platform sees agents return both "trade" and "homeowner" when only one was spoken. A major booking company loses reservations because conversations get cut off before customers finish their sentence. A customer service platform had to disable end-of-speech detection entirely because their model kept generating words that were never spoken. These aren't edge cases, they're the default experience when you bolt a generic streaming STT model onto a voice agent pipeline.

## Context carryover: what makes this different

Universal-3.5 Pro Realtime supersedes Universal-3 Pro Streaming as the recommended default for new real-time deployments, and it introduces a capability no other streaming STT provider offers yet: context carryover. The model interprets each turn in the context of prior turns in the conversation rather than transcribing every utterance in isolation, which reduces the per-turn error rate in real-world conversations with names, follow-ups, and references that only make sense given what was said a moment ago.

The mechanism is straightforward. The model takes conversation context that can be updated turn by turn. The agent's replies can be passed in at connection and refreshed mid-stream after each turn with no reconnect, giving more contextually relevant outputs. So when the agent asks "What's your email address?" the model already knows an email is coming, and transcribes "user@gmail.com" instead of "user at gmail dot com."

There are two distinct levers behind this:

- **Contextual prompting:** A natural-language description of the call, updated mid-stream via the API. The more specific the prompt, the better the accuracy. Telling the model "Cardiology consultation between Dr. Smith and an elderly patient regarding chest pain and medication adjustment" is far more useful than just "Medical call."
- **Conversation context (agent context):** The model retains prior STT transcripts within the session, and you can additionally pass in the agent's LLM-generated TTS responses, giving the model the full two-sided picture of the conversation.

Context carryover helps specifically with short answers, spelled-out entities like emails and IDs, and similar-sounding words. You can set `previous_context_n_turns=0` to disable automatic carryover. The default window is 3 prior turns, configurable up to 100.

## The benchmark picture

The model offers three default modes: Balanced (default), Max Accuracy, and Min Latency, each a combination of lower-level streaming parameters. On AA-WER Streaming, the Max Accuracy variant is level with its predecessor Universal-3 Pro Realtime, and the Min Latency variant is roughly 10% faster.

Here's how it stacks up against the competition on the Artificial Analysis streaming benchmark:

| Model | WER (Final) | Time to Final |
| --- | --- | --- |
| ElevenLabs Scribe v2 Realtime | 3.6% | 0.14s |
| Cartesia Ink-2 (external) | 3.7% | 0.09s |
| **Universal-3.5 Pro (Max Accuracy)** | **4.1%** | **0.44s** |
| Universal-3.5 Pro (Min Latency) | 4.3% | 0.40s |
| Deepgram Nova-3 Realtime | 6.6% | 0.07s |
| Deepgram Flux | 7.4% | 0.02s |

According to the Pipecat STT Benchmark, for production voice agents P95 latency is often more telling than the median. Universal-3.5 Pro Realtime measures 534ms at P95. The tradeoff is clear: ElevenLabs and Cartesia are faster to emit a final, but Universal-3.5 Pro is more accurate than anything that beats it on latency.

## What else is new

The model supports 18 languages, up from 6 in Universal-3 Pro Realtime, with mid-sentence code-switching. It also ships with Voice Focus noise cancellation. Voice Focus can be set to "near-field" or "far-field" to isolate the primary voice and suppress background noise, with a configurable suppression threshold from 0.0 to 1.0.

Other capabilities carried over from the predecessor:

- **Keyterm prompting:** Domain-specific terms fail at 3 to 5x higher rates than general speech in off-the-shelf models. Feed the model up to 1,000 words of domain vocabulary and it applies corrections as audio arrives.
- **Disfluency control:** Prompt for verbatim output (fillers, restarts, stutters) for compliance workflows, or clean output for customer-facing summaries, on the same stream.
- **Real-time speaker diarization:** The model tracks speaker turns inline, at streaming speed, with native turn detection built directly into the transcript. It understands when a speaker is done based on tonality, pacing, and speech patterns, not just silence.
- **Dynamic mid-session prompting:** Update the model prompt in real time over WebSocket as the conversation progresses without dropping the connection or restarting the session.

## Where it falls short

In Max Accuracy mode, Universal-3.5 Pro Realtime achieves 4.1% WER at 0.44s after end of speech, behind the more accurate Cartesia Ink-2 external endpoints (3.7%, 0.09s) and ElevenLabs Scribe v2 Realtime (3.6%, 0.14s). If raw latency to first final is the primary constraint, Deepgram Flux at 0.02s is in a different league, though at a significant accuracy cost. Universal-3.5 Pro is the accuracy leader among models that are competitive on speed, not the speed leader overall.

## The real shift: STT that knows what's happening

The deeper implication here is architectural. Until now, voice agent builders had to compensate for stateless STT by doing their own post-processing: regex for emails, custom entity extractors, prompt engineering on the LLM side to handle mishears. If you're building voice agents, this combination of contextual prompting and context carryover is designed for you: your application already knows who's calling, what they bought, and what the agent just said. Now your speech-to-text model can know it too.

The practical use cases where this matters most:

- **Contact centers:** Account IDs, policy numbers, and credit card details transcribed correctly on the first pass, without post-processing.
- **Medical transcription:** Medication names and dosages captured accurately when the prompt includes the clinical domain.
- **Multilingual support:** 18 languages with mid-sentence code-switching, covering real-world bilingual calls.
- **Compliance workflows:** Disfluency control for verbatim records, speaker diarization for correct attribution.

## Getting started

Universal-3.5 Pro Realtime costs $0.45/hr ($7.50 per 1,000 minutes), unchanged from Universal-3 Pro Realtime. Both Max Accuracy and Min Latency modes are priced the same. If you're already on Universal-3 Pro Streaming, upgrading is a one-line change: update the `speech_model` parameter to `u3-rt-pro`.

Here's a minimal Python setup using the AssemblyAI SDK with context carryover enabled:

```
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions,
    StreamingEvents, StreamingParameters, TurnEvent
)
def on_turn(self, event: TurnEvent):
    print(f"{event.transcript} (end_of_turn={event.end_of_turn})")
    # After agent responds, update context:
    # client.update_configuration(agent_context="What is your email address?")
client = StreamingClient(
    StreamingClientOptions(api_key=API_KEY, api_host="streaming.assemblyai.com")
)
client.on(StreamingEvents.Turn, on_turn)
client.connect(
    StreamingParameters(
        speech_model="u3-rt-pro",
        sample_rate=16000,
        continuous_partials=True,
    )
)
```

The model works natively with LiveKit, Pipecat, Vapi, and Twilio, so teams already on those stacks can drop it in without rearchitecting. The [full benchmark results](https://artificialanalysis.ai/speech-to-text/streaming) are available on Artificial Analysis, and the [contextual awareness deep-dive](https://www.assemblyai.com/blog/contextual-awareness-in-universal-3-5-pro-realtime) walks through the prompting mechanics in detail.

## Sources

- [X post by @artificialanlys](https://x.com/artificialanlys/status/2074160133702402314)
- [Artificial Analysis: Streaming STT Leaderboard](https://artificialanalysis.ai/speech-to-text/streaming)
- [AssemblyAI: Contextual Awareness in Universal-3.5 Pro Realtime](https://www.assemblyai.com/blog/contextual-awareness-in-universal-3-5-pro-realtime)
- [AssemblyAI: Universal-3 Pro Streaming release blog](https://www.assemblyai.com/blog/universal-3-pro-streaming)
- [AssemblyAI: Top APIs for real-time speech recognition in 2026](https://www.assemblyai.com/blog/best-api-models-for-real-time-speech-recognition-and-transcription)
- [Pipecat docs: AssemblyAI STT integration](https://docs.pipecat.ai/api-reference/server/services/stt/assemblyai)