Where Context Lives in a Cascading Voice Agent — and Why the STT Layer Quietly Decides Your Accuracy

digitado ⋅ 9 de July de 2026

Your agent asks, “What’s your email address?” The caller says it out loud. And your transcript comes back as “user at hack er noon dot com.”

Now the LLM has to work with that. Maybe it guesses. Maybe it asks again and annoys the caller. Either way, the mistake didn’t happen at the LLM — it happened one layer earlier, at the part of the stack most teams treat as a solved commodity: the speech-to-text.

Here’s the thing about the cascading voice agent architecture that doesn’t get said enough. You can swap in the best LLM on the market and the most natural-sounding TTS money can buy, and your agent will still feel broken if the transcript is wrong. Everything downstream is only ever reacting to text. If the text is wrong, the agent is confidently answering a question the user never asked.

So this piece is about where context actually lives in a cascading stack — because it’s not one place, and the most important one isn’t where most people put it.

What is the cascading (STT → LLM → TTS) voice agent architecture?

A cascading voice agent — also called a chained architecture — pipelines three specialized components: speech-to-text turns the user’s audio into text, an LLM decides what to say, and text-to-speech turns that reply back into audio. It’s the alternative to a single end-to-end “speech-to-speech” model that does all three inside one network.

The flow looks like this:

User audio → STT (Universal-3.5 Pro Realtime) → turn detection → LLM → TTS → user audio

The tradeoff is straightforward. An end-to-end model is simpler — one thing to call. A cascading stack is more moving parts, but you get control: you pick the best model for each job, you can swap any layer without rebuilding the others, and when something breaks you can actually see which stage did it. For teams that need accuracy, custom conversation logic, and the freedom to change vendors, cascading still wins in 2026.

Most teams don’t wire these three services together by hand, either. They run the pipeline on an orchestration framework like LiveKit or Pipecat — and AssemblyAI ships drop-in plugins for both, so the STT layer is a one-line change, not an integration project.

But orchestration only routes the audio. It doesn’t decide what your agent hears correctly. That’s the part we need to talk about.

Where context actually lives in the pipeline

“Give the agent more context” is advice that sounds obvious and means nothing until you say which context and which layer. In a cascading stack, context lives in four different places, and they’re owned by different components.

Conversation history — usually held at the LLM. This is the one everyone thinks of: the running transcript of who said what. It’s necessary, but remember the LLM only ever sees text. It inherits every error the STT made. History at the LLM can’t repair a name the transcriber never got right.

The agent’s own prior turns — fed back into the STT. This is the one that gets skipped, and it’s the highest-leverage. If your STT model knows the agent just asked “What’s your email address?”, it can anticipate the shape of the answer. That’s exactly what Universal-3.5 Pro Realtime’s agent_context parameter does. Pass the agent’s question in, and the model transcribes the reply as user@hackernoon.com instead of “user at assembly a i dot com.” Same audio. Different result. Because the model knew an email was coming.

The user’s prior turns — carried forward automatically. Universal-3.5 Pro Realtime keeps STT-finalized user turns in context on its own. You don’t configure anything for the user half of the dialog; it comes along for free.

Domain vocabulary — injected at the STT. Product names, SKUs, drug names, the weird spelling of your company. You boost these with keyterms_prompt, and you can update the contextual prompt mid-conversation through UpdateConfiguration as the call moves between stages.

Notice the pattern. Three of the four highest-value context sources belong at the speech-to-text layer — the layer teams most often pick last, by price. That’s the whole argument in one sentence: garbage in, garbage out starts at the STT. Context injected early, before the transcript is finalized, prevents the error. Context added downstream can only react to it.

Build a cascading voice agent that uses context correctly

Enough theory. Here’s the STT layer of a cascading agent on LiveKit, using Universal-3.5 Pro Realtime. It’s available on livekit-agents 1.6+ — you set the model and you’re transcribing.

from livekit.plugins import assemblyai

stt = assemblyai.STT(
    model="universal-3-5-pro",
    # Seed the model with the agent's opening line at connection time
    agent_context="Hi, thanks for calling. Can I get your email to pull up your account?",
    # Boost domain terms the model might otherwise miss
    keyterms_prompt=["AssemblyAI", "Universal-3.5 Pro", "SKU-4490", "Kelsey"],
)

The important move is keeping agent_context fresh. Seed it with the greeting at connection time, then push each new agent reply mid-stream so the model always knows what it just asked:

# After the agent speaks, update context before the user responds
stt.update_options(
    agent_context="What's the best phone number to reach you at?"
)

Now when the caller rattles off a phone number, the model is expecting digits in a phone-number shape — not a stream of words that happen to sound like numbers.

If you’re on Pipecat instead, the same model drops into Pipecat mode, where VAD and the turn analyzer drive endpointing:

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
    ),
    vad_force_turn_endpoint=True,  # Pipecat mode default
)

One honest note before you build the whole thing by hand: the cascading path is the control path. If you’d rather not manage three vendors, three bills, and three sets of logs, AssemblyAI’s Voice Agent API collapses STT, LLM, and TTS into a single WebSocket at a flat $4.50/hr — built on the same Universal-3.5 Pro Realtime underneath. Pick cascading when you need to own the pieces; pick the one API when you want to ship this afternoon.

The part everyone underinvests in: turn detection and the latency-vs-accuracy trap

Turn detection decides when the user is done talking. Get it wrong in one direction and your agent barges in over people. Get it wrong in the other and you leave dead air that makes the whole thing feel broken.

Universal-3.5 Pro Realtime handles this with punctuation-based end-of-turn detection: it watches for terminal punctuation — ., ?, ! — combined with configurable silence thresholds (minturnsilence, maxturnsilence). It’s not guessing from a confidence score. The model understands when a sentence is actually complete, and it emits fully formatted transcripts — punctuation and capitalization already there — at sub-300ms latency. No separate formatting pass, no waiting.

Here’s where teams talk themselves into a bad decision. They optimize for the lowest possible latency number and treat accuracy as a rounding error. But a fast wrong transcript is still wrong — you’ve just arrived at the incorrect answer sooner. The models that win voice agent evals do it on the boring stuff: entity accuracy on credit card numbers, phone numbers, emails, and physical addresses, plus reliable handling of the one-word utterances that actually break agents — “yes,” “no,” “mmhmm.” That’s what keeps a conversation from derailing.

Multilingual is a context problem too

Universal-3.5 Pro Realtime supports native code-switching across 18 languages within a single session, plus regional dialects out of the box. That matters for cascading stacks specifically, because a language-detection error at the STT layer cascades exactly like a transcription error does — the LLM ends up reasoning over text that’s in the wrong language or garbled at the switch point. Getting the language right early is just another form of getting the context right early.

The real axis isn’t control vs. simplicity

The cascading debate almost always gets framed as control versus simplicity — DIY the pipeline, or hand it to one model. That’s the wrong axis.

Whichever architecture you choose, your ceiling is set by the quality of what your agent hears. Context is how you raise that ceiling, and it belongs at the speech-to-text layer — early, before a bad transcript can propagate through the LLM and out through the TTS as a confidently wrong answer. Universal-3.5 Pro Realtime is built for that: the agent’s turns in context, the user’s turns carried forward, your vocabulary boosted, and turn detection that knows when a sentence is done.

Pick your architecture for the reasons that fit your team. Just don’t pick your STT last.

FAQ

What’s the difference between a cascading and an end-to-end voice agent? A cascading (chained) voice agent pipelines three specialized components — speech-to-text, an LLM, and text-to-speech. An end-to-end (speech-to-speech) model does all three inside a single network. Cascading gives you control, swappability, and easier debugging; end-to-end gives you fewer moving parts. For accuracy and custom conversation logic, cascading remains the stronger choice.

Where should conversation context live in a voice agent pipeline? In several places, but the highest-leverage spot is the speech-to-text layer. Passing the agent’s prior turns into the STT via agent_context lets the model anticipate the answer and transcribe it correctly — before any error can propagate downstream to the LLM.

What is agentcontext and when should I use it? agentcontext is a Universal-3.5 Pro Realtime parameter that feeds your agent’s spoken replies back into the transcription model, so it knows both sides of the conversation. Set it at connection time with your greeting, then update it after each agent reply via update_options() (LiveKit) or the UpdateConfiguration message on the raw streaming WebSocket. Use it whenever the agent’s question shapes the expected answer — emails, phone numbers, account numbers, yes/no confirmations.

How does Universal-3.5 Pro Realtime detect the end of a turn? It uses punctuation-based end-of-turn detection: terminal punctuation (., ?, !) combined with configurable silence thresholds (minturnsilence and maxturnsilence), rather than a confidence-score guess.

Does a cascading architecture add too much latency? Not with a streaming STT built for it. Universal-3.5 Pro Realtime emits fully formatted transcripts at sub-300ms latency, so the STT stage isn’t the bottleneck. Chasing lower latency at the cost of accuracy is a false economy — a fast wrong transcript is still wrong.

Like 0

Liked Liked