AssemblyAI's Universal-3.5 Pro Gives Voice Agents Memory Between Turns

Every streaming speech-to-text model on the market today has the same blind spot: it hears audio, transcribes it, and immediately forgets everything. The next utterance arrives with zero memory of what came before. That works fine for dictation, but it quietly breaks voice agents, where each user turn is a direct reply to something the agent just said. AssemblyAI's Universal-3.5 Pro Realtime is the first streaming STT model to fix this at the model level, and it's available now.

The problem no one had solved yet

Voice agent pipelines fail in predictable ways. A user says "user at gmail dot com" instead of "[email protected]" because the model has no idea an email was expected. A caller spells out a policy ID and the model mishears half of it. A medication name gets mangled because the model has never seen it before and has no domain context to fall back on.

A collections platform sees agents return both "trade" and "homeowner" when only one was spoken. A major booking company loses reservations because conversations get cut off before customers finish their sentence. A customer service platform had to disable end-of-speech detection entirely because their model kept generating words that were never spoken. These aren't edge cases, they're the default experience when you bolt a generic streaming STT model onto a voice agent pipeline.

Context carryover: what makes this different

Universal-3.5 Pro Realtime supersedes Universal-3 Pro Streaming as the recommended default for new real-time deployments, and it introduces a capability no other streaming STT provider offers yet: context carryover. The model interprets each turn in the context of prior turns in the conversation rather than transcribing every utterance in isolation, which reduces the per-turn error rate in real-world conversations with names, follow-ups, and references that only make sense given what was said a moment ago.

The mechanism is straightforward. The model takes conversation context that can be updated turn by turn. The agent's replies can be passed in at connection and refreshed mid-stream after each turn with no reconnect, giving more contextually relevant outputs. So when the agent asks "What's your email address?" the model already knows an email is coming, and transcribes "[email protected]" instead of "user at gmail dot com."