Phone AI agents: the architecture, and the pitfalls nobody shows in the demo

A voice agent that takes the call, books the appointment, answers the usual questions from your knowledge base and hands over to a person when needed: that is one of the most requested AI projects right now. The demo is easy. What makes it work on a real phone line is timing, interruptions, a clean hand-over and the legal basics. Here is the architecture I use and the pitfalls to plan for.

Dominik Heinrich

November 8, 202610 min read

1. Start with the job, not the voice

The best phone agents do a few things very well: answer the questions your staff answer twenty times a day, book or move an appointment, take a message, and pass everything else to a person. Write those jobs down with the information each one needs, the systems it touches and what counts as done.

That list decides the architecture far more than the choice of voice. A clinic that books appointments needs calendar access and identity checks; a restaurant that answers opening hours needs neither. Start narrow, measure, then add.

2. Two architectures: a pipeline or one speech model

The classic pipeline has three parts: speech to text, a language model that decides what to say and which tool to call, and text to speech. Each part is replaceable, you can log the text of every turn, and tool calls work exactly as in a chat agent. The cost is latency: three steps that each take time.

The newer option is one speech-to-speech model, such as Google's Gemini Live API, that listens and answers in audio directly. It reacts faster and sounds more natural, but you have less control over each step and fewer places to apply your own checks. My default for business processes is the pipeline with streaming at every step; I use a speech-to-speech model where conversation quality matters more than tight control.

3. The phone line: numbers, media streams, WebSockets

The call arrives at a phone number from a provider such as Twilio or Azure Communication Services. From there the audio has to reach your agent in real time. Twilio Media Streams sends the raw audio of a live call over a WebSocket and, in bidirectional mode, takes audio back; Twilio's ConversationRelay goes a step further and handles speech recognition and synthesis for you, so your server only sees text. Azure Communication Services offers Call Automation for the same building blocks on Azure.

Whichever you pick, keep the agent server close to the provider's media region and keep the WebSocket path short: no extra proxies, no buffering layers. Every hop adds delay you will hear.

The pipeline of a phone agent, the speech-to-speech alternative and the way out to a person. Click to open full size.

4. The latency budget: time to first audio

Telephony has long had its own standard for how much one-way delay a conversation tolerates (ITU-T G.114). A voice agent adds model time on top of the network, so treat delay as a budget you split across the steps: detecting that the caller has finished, transcribing, the model's first words, and the first audio of the answer.

The metric I track is the time from the end of the caller's sentence to the first audio we play back, on every turn, and I look at the slow turns, not the average. Three things help most: stream every step instead of waiting for whole sentences, keep spoken answers short, and start tool calls as soon as the intent is clear. If a lookup takes longer, say so ("one moment, I'm checking the calendar") instead of going silent.

5. Barge-in and turn-taking

People interrupt. They say "no, Tuesday" while the agent is still reading out Monday's slots. The agent has to notice the caller speaking, stop its own audio at once and listen. Without this barge-in, callers talk over a voice that doesn't stop, and the call falls apart.

Also design for the phone, not for a screen: never read out lists of ten options, offer two; repeat important details such as the date and time back to the caller; and avoid anything that only works visually, like links or tables. A phone conversation is short sentences and confirmation.

6. Tools and data: the same rules as your staff

The agent books, moves and cancels through your own systems, with tools that are as narrow as possible. Give it the same rules your receptionist has: check who is calling before revealing or changing anything personal, read the change back and get a yes before saving it, and never handle payment card numbers in the conversation; payments belong in a dedicated, certified flow.

Log the text of every turn and every tool call, so you can review difficult calls, improve the knowledge base and prove what the agent did when someone asks.

7. Hand-over: warm, with a summary

Every phone agent needs a way out. When the caller asks for a person, when the agent doesn't understand twice in a row, or when the topic is outside its jobs, it transfers the call. A cold transfer, where the caller repeats everything, undoes the time the agent saved. A warm transfer hands the person who picks up a two-line summary: who is calling, what they want, what has been checked.

Plan the case where nobody is free or the office is closed: take a message with a callback number and create a task, rather than ringing into the void. Twilio's Dial and the call transfer features of Azure Communication Services cover the telephony side.

8. Consent, recording and saying it's an AI

Open every call by saying the caller is talking to an AI assistant. In the EU this becomes a legal duty: the AI Act requires that people are informed when they interact directly with an AI system, and that obligation applies from 2 August 2026.

Recording rules differ by country, so get advice for the markets you serve. A safe default: announce recording at the start, record only when you need it, keep transcripts instead of audio where that is enough, set a retention period and keep personal data out of logs that don't need it. And give the caller a way to reach a person at any point.

Takeaway
A phone agent succeeds on timing and trust: stream everything, let callers interrupt, act with your staff's rules, hand over warm and say clearly that it's an AI.

Real projects behind this

Booking & operations platform for hospitality

Try it yourself

Production readiness

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides