7 Things You Can Build With a Single WebSocket (Using AssemblyAI’s Voice Agent API)
Most voice AI architectures look like a Rube Goldberg machine. You pipe audio into a speech-to-text service, feed the transcript to an LLM, send the LLM’s reply to a text-to-speech engine, then duct-tape the audio back to the user. Each hop adds latency, failure modes, and billing dashboards.
AssemblyAI’s Voice Agent API collapses all of that into one WebSocket connection. You stream mic audio in, you get spoken agent responses back. Turn detection, tool calling, barge-in—it’s all built in. The endpoint is wss://agents.assemblyai.com/v1/ws.
Here are seven things you can build on top of it—each one surprisingly little code.
1. A multilingual support line that switches voices mid-call
The Voice Agent API ships with 18 English voices and 16 multilingual voices that support code-switching between languages and English. That means your agent can greet a caller in English, detect that they’d prefer Spanish, and swap to the lucia voice with a single session.update message—no reconnection, no new session.
The config is dead simple:
{ "type": "session.update", "session": { "output": { "voice": "lucia" } } }
You could wire this up with a language-detection tool: register a tool called detect_language, and when the agent invokes it, respond with the detected language and fire a session.update to change the voice. The user never notices a seam.
2. A voice-powered knowledge base you can actually talk to
The API’s tool calling feature lets you register external functions that the agent can invoke mid-conversation. Register a search_docs tool that hits your docs index (Pinecone, Elasticsearch, whatever), and suddenly you have a voice interface to your entire knowledge base.
The quickstart in AssemblyAI’s docs actually ships with an MCP server wired up this way. You ask a question out loud, the agent calls the tool, gets the answer, and speaks it back—all over that single WebSocket. No reading docs required, ironically.
3. A real-time language tutor that code-switches naturally
The multilingual voices don’t just speak one language—they support code-switching. The arjun voice handles Hindi/Hinglish and English natively. pierre does French and English. That’s exactly what a language tutor needs: the ability to drop in and out of the target language mid-sentence.
Set your system prompt to something like: “You are a French tutor. Speak mostly in French but switch to English to explain grammar. Correct the user’s pronunciation gently.” Pair it with the pierre voice and you’ve got a conversational language partner that’s available 24/7.
4. A voice-driven order system for restaurants
Phone ordering is still massive in food service, and most of it is still handled by humans. Build an agent with tools like getmenu, addtoorder, and submitorder that hit your POS API. The agent takes the order conversationally, confirms items, and submits it—all while the kitchen staff keeps working.
The built-in turn detection with adjustable VAD threshold means the agent won’t cut the customer off mid-sentence when they’re reading a complicated order. And barge-in support means they can interrupt to say “actually, make that a large” without waiting for the agent to finish talking.
5. A meeting copilot you can interrupt and question
Most meeting transcription tools are passive—they record and summarize after the fact. But what if you could talk to the transcript in real time?
Feed meeting audio into the Voice Agent API and register tools like searchtranscript and getaction_items. During the meeting, you can ask “What did Sarah say about the deadline?” and get a spoken answer. Session resumption (sessions persist for 30 seconds after disconnection) means you don’t lose context if your connection hiccups.
6. A browser-based voice concierge for SaaS onboarding
The API has a clean browser integration path: your server mints a short-lived token via GET /v1/token, and the browser opens the WebSocket with that token as a query param. Your API key never touches the client.
Embed this in your app’s onboarding flow and you’ve got a voice concierge that walks new users through setup. Register tools like createproject or inviteteammate so the agent can actually perform actions in your app while talking the user through it. It’s like having a customer success rep embedded in your UI—except it scales to every user simultaneously.
7. A lead qualification agent that routes to sales
Inbound sales calls are time-sensitive. Every minute a lead waits, your close rate drops. Build an agent that picks up immediately, asks qualifying questions conversationally, and uses a routetosales tool to hand off warm leads to the right rep—complete with a transcript summary.
The system prompt does the heavy lifting here: define your qualification criteria, tell the agent which questions to ask, and specify when to escalate. The agent handles the rest. Because it’s all one WebSocket, the latency between the caller saying something and the agent responding is minimal—no awkward silence while three different services talk to each other behind the scenes.
The common thread
Every one of these projects would’ve been a multi-service integration nightmare a year ago. The Voice Agent API compresses the entire voice AI stack—speech recognition, language understanding, voice synthesis, turn management—into a single WebSocket you can connect to in about 50 lines of code.
The interesting part isn’t any one of these use cases. It’s that the same API handles all of them. Swap the system prompt and tools, and the same connection becomes a completely different agent.
If you want to try it yourself, the Voice Agent API docs have a working quickstart you can run in under five minutes. And if you don’t have an API key yet, grab one for free.
