Latency in AI voice calls: why half a second matters

Updated 2026-09-19 · 4 min read · By the Convo360 team

What 'latency' means on a voice call

On a phone call, latency is the silence between the caller stopping talking and the agent starting to respond. In face-to-face conversation this gap is usually under half a second. Any longer and it starts to feellike the other person wasn't listening, got distracted, or the line has dropped.

For an AI voice agent, that half-second budget has to cover several distinct steps happening one after another, not just 'the AI thinking'. Understanding where the time actually goes is what lets you fix it. Treating it as one mysterious black box makes it much harder to improve, because you end up guessing at the wrong lever.

This matters more in voice than in any other channel Convo360 supports. A WhatsApp reply that takes three or four seconds feels completely normal to a customer glancing at their phone. The exact same three or four seconds of silence on a live phone call feels broken, and many callers will simply hang up or start talking again, confusing the agent further.

The four places delay comes from

• Speech-to-text (STT): converting the caller's spoken words into text the AI can read. Fast engines do this in a fraction of a second; some add a deliberate pause first to make sure the caller has actually finished talking before committing to a transcript.
• The LLM's response time: the language model has to read the conversation and generate a reply. Larger, more capable models are often slower per response than smaller ones, and this is usually the biggest single chunk of the delay, and the one most within your control.
• Text-to-speech (TTS): turning the generated reply back into audio. Some providers start streaming audio as soon as the first few words are ready rather than waiting for the whole sentence to be generated, which noticeably reduces perceived delay even if the total processing time is similar.
• Network and telephony: the call itself travels over the telephony provider's network and the customer's own connection. A poor mobile signal or a call from a basement parking area adds delay no software setting can fix.

Each of these adds up in sequence, not in parallel. A system that's individually 'fast enough' at each stage can still feel sluggish once you stack all four together, which is why fixing just one stage, say switching to a faster LLM, sometimes produces a smaller improvement than expected if another stage is quietly slow too.

Why the silence-before-reply setting matters

There's a genuine trade-off buried in the first stage: reply too fast and the agent talks over a caller who paused mid-sentence to think; wait too long and the call feels laggy and unresponsive. Convo360's Call behaviour section exposes silence-before-reply timing under Advanced, alongside Background noise handling and Let the caller interrupt, so you can tune this for how your specific customers speak. A quick, clipped conversation style needs a shorter pause than a caller who thinks out loud or pauses to check a document while talking.

This setting interacts with Background noise handling too: on a noisy line, a slightly longer pause combined with Noisy mode and interruptions turned off stops background sound from being mistaken for the caller starting to speak again, which would otherwise cut the agent off mid-sentence for no real reason.

Acknowledgement phrases hide the gap

Even with a fast setup, there's usually a brief moment where the agent is still 'thinking', retrieving from a knowledge base, or waiting on the model to finish generating. A short, natural acknowledgement, 'sure', 'haan', 'one moment', spoken immediately, before the full answer is ready, fills that silence and keeps the call feeling responsive, even if the actual answer takes another few hundred milliseconds to generate.

Convo360's Acknowledgement style setting lets you choose a natural short word, a fuller phrase, none at all, or write your own custom line. This single setting often does more for how 'natural' a call feels than any raw speed optimisation, because it changes the caller's perception of responsiveness even when the underlying processing time hasn't changed at all.

Choosing providers and testing latency

Provider choice affects latency more than most other decisions. Since Convo360 is BYOK, you choose your own STT/LLM/TTS combination per agent: OpenAI, Anthropic Claudeor Google Gemini for the language model, and Sarvam, Smallest.ai, ElevenLabs or OpenAI for voice. A lighter, faster model generally replies quicker than the largest available one, which matters more for voice than for chat, where a reader tolerates a few extra seconds far more easily than a caller does on a live line.

Test on a real phone call over a real mobile network, not just in a text-based preview. Network conditions and actual audio streaming behave differently from a typed test message, and issues like echo, jitter or a weak signal only show up on an actual call. Call your own number, note where the pauses feel longest, and adjust the model, voice provider or acknowledgement setting accordingly. Review Flow Logs after a batch of test calls to spot any consistently slow step, and repeat the test after any change rather than assuming a fix worked.

Step by step

1. Pick a faster model for voiceChoose a lighter, faster LLM for the voice agent rather than defaulting to the largest available model.
2. Turn on an acknowledgement phraseSet Acknowledgement style to a short natural word so the caller hears a response immediately while the full reply generates.
3. Tune silence-before-replyAdjust the Advanced silence timing in Call behaviour to match how quickly your typical caller finishes a sentence.
4. Test on a real phone callCall the live number over a normal mobile connection and note where pauses feel longest, rather than relying only on a text preview.
5. Review Flow Logs for slow stepsCheck call logs after test calls to spot which stage — STT, LLM or TTS — is consistently adding delay.

Example conversation

Frequently asked questions

Related reading

Available across India — see WhatsApp & AI calling automation by state and city.