AI voice agents, and the parts that decide whether they work

Timing is most of the experience

Text chat forgives a pause. A phone call does not. When a caller stops speaking, silence acquires a meaning, and if the reply takes long enough to feel like thinking, people repeat themselves, talk over the answer, or assume the line has dropped.

That delay is not one thing. Speech has to be recognised, the system has to judge whether the caller has finished or is only drawing breath, the model has to produce a reply, the reply has to become audio, and all of it travels a network path that may leave the country and come back. Every stage adds, and the one that usually costs the most is the judgement about whether the caller is done.

• Speech recognition running while the caller speaks rather than after they stop
• End-of-turn detection, which is a judgement rather than a measurement
• The reply itself, kept short, because a long answer takes longer to begin
• Speech synthesis streamed, so the first words play before the sentence is finished
• Network distance, including where the model and the telephony provider are hosted

Any lookup the agent makes is heard by the caller as silence, which is why a slow internal system produces a slow voice agent no matter which platform is underneath it.

A large share of the demand for this arrives under a different name. People searching for an AI receptionist are describing the same technology pointed at a front desk, and the questions they ask first are cost, whether a caller has to be told it is a machine, and what happens when it cannot understand somebody. Those are answered on that page. What follows here is the engineering underneath.

Interruption, accents and how people actually talk

Real calls do not resemble a script. People interrupt, correct themselves halfway through a sentence, read a number out with pauses in the middle, and say things like no, the other one. A voice agent that cannot be interrupted becomes an obstacle almost immediately, and one that treats every pause as a finished sentence will cut people off while they are still thinking.

Accent and language mixing are the other half of it. Across South Asia and the Gulf a caller will begin a sentence in English and finish it in another language without noticing they have done it, and names, addresses and place names are precisely the words a recogniser handles worst. Designing for that is not a refinement, it is the difference between a system that survives real calls and one that only works in a demonstration.

• Barge-in enabled, so speaking over the agent stops it mid-word
• Critical values read back for confirmation: numbers, dates, spellings, amounts
• A correction path, because no, not that one is a normal thing for a person to say
• Recognition tuned with the vocabulary the business actually uses: product names, area names, brand names
• A fallback that offers to send a message or a link rather than asking a caller to repeat a long detail a third time

What a voice agent should hand over, and how

The handover gets designed last and decides how the whole system is remembered. A caller who has explained their problem twice, once to a machine and once to a person, has had a worse experience than if they had simply waited on hold.

Handover rules belong in the build rather than in the model's judgement. Money in dispute, a complaint, a cancellation, or any promise the business would be held to goes to a person. So does any caller who asks for one, immediately, without being talked out of it.

• Transfer while the caller is still on the line wherever somebody is available, rather than promising a call back
• The transcript and the reason for transfer delivered to whoever takes over, before they speak
• A defined behaviour outside working hours, usually a callback booked into a real calendar slot rather than a vague promise
• An escalation phrase that always works, tested regularly, because it is the thing that rescues a bad call

Two counts describe a voice deployment better than any single success rate: the handovers a person had to redo, and the calls that should have reached a person and never did.

Telephony or WhatsApp, and why the answer changes by country

There are two ways to give an agent a voice, and the right one depends on where the caller is. Ordinary telephony means obtaining a number in the market and connecting it to a carrier such as Twilio or Telnyx, usually through an orchestration layer such as Vapi that handles turn-taking, interruption and tool calls. The other route is voice inside WhatsApp, which behaves as an internet call and needs no local number at all.

Meta publishes country rules for the second route, and they are asymmetric in a way that catches people out. Business-initiated calling on the WhatsApp Business Calling API is excluded in the United States, Canada, Egypt, Vietnam and Nigeria, along with sanctioned regions. Pakistan is not on that exclusion list. Calls the customer places are supported wherever Meta business messaging operates. That is a platform rule rather than something any vendor arranges around, and Meta revises the list, so the current documentation is worth checking before a design depends on it.

The consequence is that two markets can need opposite architectures. A market excluded from business-initiated WhatsApp calling has to reach customers over telephony. A market where obtaining and operating local numbers is the harder problem can lean on WhatsApp instead. One design applied to both will fail in one of them, usually the one nobody tested.

Consent, recording and disclosure

Voice attracts more regulation than text, and most of it is local. Recording consent differs by jurisdiction, and in several places every party on the call has to agree before a recording starts. Automated outbound calling is separately regulated in many markets, with rules on calling hours, do-not-call registers and prior consent, and those rules apply to a machine placing the call exactly as they apply to a person.

Two things are worth treating as fixed regardless of jurisdiction. Say that it is an automated system, early and plainly, because a caller who works it out mid-complaint feels deceived and will say so publicly. And never use a voice cloned from a real person without their written permission, staff included.

• Disclosure at the start of the call, in the language the call is being conducted in
• Recording consent handled to the standard of the strictest market you operate in
• Opt-out honoured on the call itself rather than through a form somewhere afterwards
• Retention limits set for recordings and transcripts, and actually applied
• Outbound calling checked against local rules on hours, consent and registers before the first call goes out

Where a voice agent is the wrong answer

If call volume is modest and somebody answers within a couple of rings, a voice agent adds a layer without removing a cost. What earns its place is volume, coverage outside working hours, and the queue at a predictable weekly peak that nobody wants to staff for.

If most calls are complaints or otherwise emotionally loaded, automation is the wrong place to meet the caller. Those calls decide whether somebody stays a customer, and the saving is trivial next to what is being risked.

If the systems the agent must query are slow, the caller hears every second of it, and the honest fix is the system underneath rather than the voice layer on top. And if the intention is calling a list that never asked to be called, this is not worth buying from anyone, because the regulatory exposure is real and the results are poor.

What moves the number, how soon it runs, and who owns it afterwards

No figure is published for voice work, and what moves the number here is unusual enough to be worth listing. Talk time is the meter on almost every platform underneath, so your weekly minutes and your worst hour matter more than your headcount. Then the number of lookups the agent makes during a call, because every one of them is heard by the caller as silence and shortening it is engineering rather than a setting. Then the count of markets, since a number in each country carries its own carrier charges and its own rules about automated calling. Then who is answerable in month four, when a provider changes a model's behaviour and the pauses lengthen without anybody having touched the build.

The audit takes the first week and on voice most of it goes on the two decisions that are expensive to reverse: which route the calls take, telephony or WhatsApp, and where the model and the telephony provider sit relative to the caller. One agent doing one job end to end is live inside a fortnight, tested against recordings of real callers rather than against your own staff, who all speak the way the person configuring it expects them to. Routing between two agents comes later, and only once the single one has held under a real week.

What you own at the end is the apparatus rather than permission to use it. Numbers are bought in your name. The prompts, the tool definitions, the transcripts and the call recordings sit in accounts under your logins. The orchestration is a set of files rather than a configuration living inside somebody else's product. If you leave, a running agent leaves with you, and so do the transcripts, which matters because they are the only honest record of how the thing actually behaves rather than how it demonstrated.

Building one is genuinely possible now and plenty of people have. The build is not the difficulty: a carrier number, a speech layer, a model and an orchestrator will produce something convincing over a weekend. The difficulty is whether somebody in-house will own the two counts every week, the handovers a person had to redo and the calls that should have reached a person and did not. Where that person exists, keep it. Moiz Khan owns automation architecture at Wobble and was Director of AI at a United States real estate company before this, leading the implementation of AI callers for working agents. The company is based in Karachi, bills month to month, and runs more than 500 calls a day for Big Texas Land Buyers.

Common questions

What is an AI voice agent?

An AI voice agent is a system that answers or places phone calls, understands what the caller says, looks things up in your systems and speaks a reply. Unlike a phone menu it does not ask the caller to choose from options, and unlike a chatbot it has to cope with interruption, background noise and the timing of a real conversation.

How fast does a voice agent need to respond?

Fast enough that the pause after the caller stops speaking feels like a normal conversational gap rather than thinking time. The delay is the sum of speech recognition, deciding the caller has finished, generating the reply, producing audio and network distance. Any database lookup is added on top, which is why slow internal systems produce slow voice agents.

Can an AI voice agent handle interruptions and different accents?

Interruption has to be supported deliberately, so that speaking over the agent stops it mid-word. Accents and language mixing are handled by tuning recognition with the vocabulary the business actually uses, including product and place names, and by reading critical values back for confirmation rather than acting on something heard once.

Can AI voice agents call customers on WhatsApp?

The platform rules decide that by country. Business-initiated calling on the WhatsApp Business Calling API is excluded in the United States, Canada, Egypt, Vietnam and Nigeria and in sanctioned regions, and Pakistan is not on that exclusion list. Calls placed by the customer are supported wherever Meta business messaging operates. Meta revises the list, so check the current documentation before designing around it.

Is it legal to record calls handled by an AI agent?

It depends on the jurisdiction, and in several places every party must consent before recording begins. Automated outbound calling is separately regulated in many markets, covering calling hours, prior consent and do-not-call registers, and those rules apply to a machine placing the call exactly as they do to a person. Disclosing that the caller is speaking to an automated system is worth treating as standard everywhere.

What should a voice agent hand over to a person?

Money in dispute, complaints, cancellations, anything the business would later be held to, and any caller who asks for a person. The transfer carries the transcript and the reason so the caller never repeats themselves, and where nobody is available the fallback is a callback booked into a real slot rather than a promise to ring back.