Voice Agent Playbook: What Production Would Require
TL;DR
- I prototyped voice reception on Retell AI and Twilio for beauty and trade workflows. I did not ship it to production
- The model is 10% of what makes a voice agent work in production; conversation flow design, latency management, and graceful human handoff are the other 90%
- Voice agents work best for structured, high-volume inbound calls (bookings, enquiries, triage) and fail at open-ended conversations requiring empathy or complex negotiation
The caller should know they are talking to an AI and still be able to complete a straightforward task without friction.
That is the useful bar. The caller has a straightforward need, such as booking an appointment or asking about availability, and the system handles it safely without pretending to be human.
I built and tested voice prototypes for OpenChair and OpenTradie. Neither was deployed to businesses or used for production calls. The work was valuable because it exposed what a production decision would require: narrow authority, explicit disclosure, reliable tool execution, low latency and a credible human handoff.
Building a voice agent that works in a controlled test is straightforward. Deciding that it is safe and useful enough for a real customer is a different product problem. I held voice back while prioritising the booking, management and communication workflows operators needed first.
Why voice, why now
Phone calls remain the primary customer contact channel for local services. Salons, tradies, medical practices, restaurants. These businesses live and die by the phone, and most of them miss 20% to 40% of inbound calls because the owner is with a client, on the road, or eating lunch.
Every missed call is a missed booking. For a salon averaging $80 per appointment, missing 5 calls a day is $400 in lost revenue. Per day. For a plumber charging $150 per callout, missing 3 calls a day is $450. The maths is painful and obvious.
The existing solutions are voicemail (customers hang up), call centres (expensive, generic, no access to booking systems), and "we'll call them back" (by which time they've called your competitor). An AI voice receptionist that answers every call, 24/7, with access to the booking calendar and the ability to actually schedule appointments, changes the economics of running a service business.
The technology is ready. Retell AI handles the speech-to-text and text-to-speech pipeline with low enough latency to feel conversational. Twilio handles telephony. The missing piece isn't the technology. It's the product design.
How much latency can a voice agent tolerate before callers drop off?
In a chat interface, 2 seconds of response time is acceptable. In a phone conversation, 2 seconds of silence is an eternity.
Human conversation has a rhythm. Turn-taking gaps average 200 to 300 milliseconds. When the gap stretches beyond 500 milliseconds, the caller perceives something is wrong. They repeat themselves. They say "hello?" They get frustrated. By 1.5 seconds, you've lost them.
The voice agent latency stack looks like this:
- Speech-to-text: 100 to 300ms (depends on the STT engine and whether you're streaming or waiting for the full utterance)
- LLM inference: 200 to 800ms (depends on the model, prompt length, and whether you're using a reasoning model)
- Text-to-speech: 50 to 200ms (most modern TTS engines are fast)
- Network round trips: 50 to 150ms (accumulated across the pipeline)
Total: 400ms to 1,500ms. The difference between the low end and the high end is the difference between "feels natural" and "feels broken."
The product decisions that control latency:
Model selection matters more for voice than for most AI interfaces. A reasoning model that produces a slightly better response in 800ms can lose to a faster model that produces a good-enough response in 200ms. In the prototype, I used lighter models for the conversational layer and reserved heavier reasoning for the decisions that needed it.
Streaming is non-negotiable. Wait for the full LLM response before starting TTS, and you've added 300 to 500ms to every turn. Stream the LLM output directly into the TTS engine, and the voice starts speaking while the model is still generating. The caller hears the beginning of the response almost immediately.
Pre-computation saves critical milliseconds. If the caller says "I'd like to book a haircut," the agent needs to check availability. Don't wait for the availability lookup to complete before responding. Start speaking ("Let me check what's available for you...") while the system queries the calendar in the background. This is natural human behaviour (people say "let me look at that" while they're looking), and it masks latency.
Conversation flow is product design, not prompt engineering
The biggest mistake I see in voice agent implementations is treating the conversation as a single long prompt. "You are a receptionist for a hair salon. Answer customer questions and book appointments."
That works in a demo. In production, it produces an agent that meanders, asks unnecessary questions, forgets context mid-conversation, and doesn't know when to give up and hand off to a human.
Before production, voice agents need structured conversation flows with defined states, transitions and exit conditions. The flows should be flexible, with guardrails, rather than rigid IVR-style decision trees. The handbook chapter on AI UX and interaction design covers the broader principles.
Opening: Greet the caller. Identify the intent. This needs to happen in the first 10 seconds. If the agent is still asking clarifying questions at the 30-second mark, the caller is already annoyed.
Information gathering: Collect what's needed to fulfil the intent. For a booking: service type, preferred date/time, any provider preference. Ask one question at a time. Don't front-load three questions in a single turn ("What service would you like, what day works for you, and do you have a preferred stylist?"). That works in text. It overwhelms in voice.
Confirmation: Repeat back the key details. "So that's a cut and colour with Sarah on Thursday at 2pm. Does that sound right?" This is where errors get caught. Skip it and you'll book wrong appointments.
Edge case handling: The caller wants a service you don't offer. The requested time isn't available. They want to speak to the owner about a complaint. Each of these needs a defined path, not an open-ended "figure it out" prompt.
Handoff: When the conversation exceeds the agent's scope, transfer to a human cleanly. "Let me connect you with Sarah directly for that. One moment." Not "I'm sorry, I'm not able to help with that." The first is service. The second is a dead end.
I prototyped conversation flows for two different verticals. Beauty and wellness requests are often scheduled and preference-driven. Trade enquiries can be urgent and problem-driven. The emotional register is different, so the flow, authority and handoff rules need to change with the domain.
The handoff is the hardest part
Handling the happy path in a prototype is straightforward. A graceful handoff for exceptions is where the production standard becomes much harder.
The failure modes:
Silent transfer. The agent routes to a human without context. The caller has to explain everything again. This is worse than no AI at all, because the caller already invested time in the AI conversation and now has to repeat it.
Abandoned transfer. The agent tries to transfer, but no human is available. The caller gets voicemail after a 2-minute AI conversation. Infuriating.
Scope creep. The agent tries to handle something it shouldn't. A complaint. A pricing negotiation. A complex rescheduling with multiple dependencies. The agent produces a response that sounds confident but is wrong, and the business doesn't find out until the customer shows up at the wrong time.
The fix is a combination of clear scope boundaries (the agent knows exactly what it can and can't handle), context-rich handoff (the human receives a summary of the conversation before they pick up), and fallback design (if no human is available, the agent takes a message with a specific callback commitment rather than leaving the caller stranded).
The handoff design deserves as much product attention as the happy path. Maybe more, because the handoff scenarios are the ones where the customer is already slightly frustrated.
When voice agents work (and when they don't)
Voice agents work well for:
- High-volume, structured inbound calls (bookings, availability enquiries, appointment confirmations)
- After-hours coverage (calls that would otherwise go to voicemail)
- Triage and routing (qualifying the call and connecting to the right person)
- Information lookup (hours, directions, service descriptions, pricing)
Voice agents struggle with:
- Emotional conversations (complaints, disputes, bad news)
- Complex negotiations (pricing discussions, custom quotes with many variables)
- Multi-party calls (conference calls, translator-mediated conversations)
- Conversations requiring deep domain expertise (medical triage, legal advice)
The pattern from boring agents that work applies directly, and the full catalogue of agentic AI patterns in the handbook covers how to structure these workflows: target patience-heavy tasks, not judgment-heavy ones. A booking call is patience-heavy (structured, repetitive, follows a known pattern). A complaint call is judgment-heavy (requires empathy, context, authority to make decisions). Build agents for the first category. Keep humans on the second.
The production checklist
Before shipping a voice agent, verify:
- End-to-end latency under 800ms for 90th percentile responses
- Conversation flow tested with 50+ real scenario variations (not just the happy path)
- Handoff tested for every scope boundary (what happens when the agent reaches its limit?)
- Fallback tested for no-human-available (voicemail with context, callback commitment)
- Monitoring in place for conversation completion rate, handoff rate, and customer satisfaction
- Caller consent and compliance verified for your jurisdiction (recording laws, AI disclosure requirements)
The voice agent that earns trust is the one that handles the routine brilliantly and hands off the exceptions gracefully. Both halves matter equally.
Frequently Asked Questions
Do callers need to be told they're talking to an AI?
This varies by jurisdiction and is evolving. Check the rules that apply to the business and call location. My product position is to disclose that the caller is speaking with an AI assistant and set an appropriate expectation at the start.
What's the cost per call for a voice agent?
For a typical 2-minute inbound booking call: STT, LLM inference, and TTS combined run $0.03 to $0.10 depending on model choices and conversation length. Twilio charges $0.01 to $0.02 per minute for voice. Total: roughly $0.05 to $0.15 per call. Compare that to a human receptionist at $25 to $35 per hour handling 15 to 20 calls per hour ($1.25 to $2.30 per call). The economics are compelling for high-volume call patterns.
Can you use open-source models to reduce voice agent costs?
Yes, and the quality gap is closing. Open-source STT (Whisper) and TTS (Piper, Coqui) models are viable for many use cases. The tradeoff is typically higher latency or lower naturalness compared to commercial APIs. For cost-sensitive deployments with high call volume, self-hosted models can reduce per-call costs significantly. For quality-sensitive deployments, commercial APIs are still worth the premium.
Logan Lincoln
Head of Product based in Currumbin Waters, Australia. Led a $70M regulated B2B SaaS portfolio at Cotality, then founded OpenChair and took its product strategy from market entry into paid production use.
