Your Voice Agent Transferred the Call. What Did the Next Agent Receive?
Smooth call transfers can hide two gaps: what the next agent knows, and who handles a failed customer request.
“I’ll connect you with our Billing voice agent — they’ll have everything you shared.”
That is exactly what a customer wants to hear after explaining a damaged order. They do not want to repeat their order number, their problem, or their frustration to a second voice agent.
But “everything” hides two different questions. What information will the next voice agent actually receive? And if that agent’s tool fails, who is responsible for getting the customer to a reliable next step?
A transfer button and a tool-call surface can make those responsibilities look settled. They are not necessarily settled. A handoff can change who speaks without choosing what customer history reaches the next agent. A tool call can fail without anyone having decided how to retry, update the customer, or escalate.
Voice handoffs make this especially easy to miss: a natural-sounding transfer can feel like a clean boundary even when the underlying conversation continues.
TL;DR:
I used a customer-service prototype to inspect two role-change mechanisms: ElevenLabs’ explicit agent-to-agent transfer and OpenAI Realtime’s live-session updates. In the tested paths, the handoff changed who spoke but did not choose what customer information the next agent received; neither error-path configuration showed a platform-owned plan for what happened after a tool call failed.
The practical takeaway is simple: treat agent role, customer context, and tool recovery as separate responsibilities — choose the context before the handoff and put recovery around the tool call.
Why this matters in customer service
In a customer-service call, continuity is valuable: a customer should not need to repeat an order number or explain a damaged item twice. But continuity and controlled disclosure are different requirements. A Billing agent may need the order number and a short damage summary; they may not need the full intake transcript or every detail the customer shared before transfer.
The same distinction appears when a tool fails. The customer does not care whether the failure came from a CRM, order system, or refund endpoint. They care whether the voice agent gives an accurate update, whether the request is safely retried, and whether a human or callback path exists when it cannot be resolved. Those are product and trust decisions, not merely API details.
For customer-facing voice deployments, reliability is not just whether the model gives a good reply. It is deciding what context crosses a handoff and who owns recovery when an external tool fails.
A simple model: role, context, and recovery
A production voice transfer has three separate layers:
Agent role → who speaks and what that agent is allowed to do
Context boundary → what the next agent is allowed to receive
Recovery boundary → how a failed external dependency is retried, handled, or escalated
Platforms often make the first layer visible: agent names, prompts, transfers, and tools. The other two layers may be available through configuration or may require application code. The important point is not that every platform lacks them. It is that a visible role change is not evidence that they exist or meet your requirement.
What I needed to verify
The prototype simulated a customer reporting a damaged order and asking for a refund. A Front Desk agent collected the customer’s name, order number, reason for calling, and frustration, then handed them to a Billing agent for a refund check.
The Billing agent’s refund-check tool asked the order system whether a refund was available. The test flow:
Customer reports a damaged order and asks for a refund
↓
Front Desk agent collects name, order number, reason, and sentiment
↓ handoff
Billing agent uses the earlier context and checks refund eligibility
To keep the state test focused on the handoff rather than speech recognition or turn timing, I ran the ElevenLabs path as a live voice handoff and sent the same customer turns as text into one OpenAI Realtime session. That let me inspect what happened to existing conversation context when the Billing agent took over. A corresponding ElevenLabs text run produced the same state and recall outcome. The goal was to inspect these role-change mechanisms, not to compare two audio pipelines or declare a vendor winner:
Does the handoff choose what information the Billing agent gets?
Billing agent accessed every customer fact tested after handoff. The traces did not establish a deliberately filtered, field-level boundary.
When the tool asked the order system and got an error back, who decided what happened next?
Neither observed configuration demonstrated a platform-level retry policy for the external dependency.
How to read these results: This is one two-agent, single-hop context-handoff test, not a comparison of audio pipelines. It does not evaluate microphone capture, ASR, interruption, or latency.
Finding 1: the handoff passed context, but did not choose what Billing agent received
Question: Does the handoff choose what information the Billing agent gets?
Answer: Not in these tests. The Billing agent received every customer fact I checked: name, order number, damaged-item problem, and frustration. I found no step that chose which of those facts Billing agent should receive.
On ElevenLabs, the live voice trace kept the Front Desk agent and Billing agent in one conversation. The transfer record named the next agent but did not list approved customer fields. On OpenAI, I changed the same Realtime session from Front Desk agent to Billing agent; it retained 14 earlier conversation items. In both cases, Billing agent used the earlier order number and recalled the other tested facts.
Finding 2: the tool could check the order, but the platform did not handle a failed tool call
Question: When the tool, refund system, returns an error, who owns the next step?
Answer: In the configurations I tested, the platform did not demonstrate that it owned the next step. The Billing agent could use the refund-check tool to ask the order system about a refund. But when the order system returned an error, the tested configurations did not show a built-in plan to retry, use a fallback, update the customer, or escalate.
The error checks were different: ElevenLabs used a live 503 in a combined single-agent voice configuration; OpenAI returned a synthetic error in a text-injection configuration. Neither configuration exposed a platform-owned retry policy for the refund dependency. Switching to a backup AI model when its provider fails is a different thing from retrying the customer’s refund system.
The two controls this call flow is missing
Finding 1 showed that the Billing agent got the customer’s history, but no one chose which parts it should receive. Finding 2 showed that the Billing agent could call the refund system, but no one owned what happened after an error. So I would add two controls: one for handoff context and one for recovery.
Here is how I would add them to the Front Desk → Billing flow:
Front Desk call transcript + collected customer state
↓
Choose and authorize handoff context
↓ (approved summary, permitted fields, transfer reason)
Billing voice-agent context
Billing agent’s refund-check request
↓
Recovery layer: timeout → retry rules → fallback → clear customer update → human/callback escalation
↓
Auditable result returned to the voice agent
The two controls I would add to this call flow. This is a design response to the findings, not something the prototype tested.
Context control. Before the Billing agent sees anything, select the order number, damage summary, sentiment, and transfer reason it needs. Keep the raw intake transcript under separate access and retention rules.
Recovery control. Put timeout, safe retry, fallback, and human or callback escalation in a tool wrapper, gateway, workflow engine, or service adapter. The Billing agent receives a clear result and gives the customer an honest update.
For this call flow, the ownership choices are:
- Only approved customer context reaches the next agent. An agent transfer is the visible primitive; a context-selection and authorization control owns the decision.
- A tool failure is handled predictably. A tool call is the visible primitive; a recovery layer owns timeout, retry, fallback, and escalation.
- Transfer and tool decisions can be audited. The visible primitive is a transfer event; platform and application logging need an explicit owner.
- The customer gets a clear status and escalation path. The visible primitive is the voice response; the conversation workflow and escalation design own the experience.
Before you ship: verify the two decisions the platform will not make
- What customer context will you deliberately pass to the next agent, and where will you select it?
- When a tool call fails, where will retry, fallback, customer updates, and escalation live?
A handoff changes who speaks; it does not choose what customer context passes. Choose that context before the handoff.
A tool call can fail; put recovery in the tool integration, a gateway, or the surrounding workflow.
These answers may come from a platform setting, a gateway, a workflow engine, or a dedicated service. They could also change with a multi-hop flow, a transfer mode that passes selected variables, or a deliberately constructed isolation layer.
Inspect the handoff trace. Force a realistic tool call failure. Write down who owns context selection, retries, customer updates, and escalation before a customer is left waiting or asked to repeat themselves.
Platform documentation
Your Voice Agent Transferred the Call. What Did the Next Agent Receive? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.
