📞 AI voice calling

Why AI voice agents mishear Indian names, places and numbers, and how to fix it

Updated 2026-10-22 · 4 min read · By the Convo360 team

In short

AI voice agents mishear Indian names, localities and numbers mainly because phone audio is low quality, many names are rare in training data, and callers switch languages mid-sentence. Fix it by choosing speech recognition built for Indian telephony, reading key details back for confirmation, using keypad entry for long numbers, and putting local names in the agent's knowledge base.

Key takeaways

Phone calls carry far less audio detail than a studio recording, so a model that sounds perfect in a demo can struggle on a call.
Names, localities and brand terms are the words most often misheard, and also the ones that matter most.
Reading the detail back and asking for a yes catches most errors before they become wrong bookings.
Long numbers such as order IDs and pin codes are often more reliable by keypad than by voice.
Check transcripts weekly: they show exactly which words your agent keeps getting wrong.

Where mishearing actually comes from

When an AI voice agent books "Kandivali" as "Candy Wally" or writes 98201 as 98021, the cause is almost always the speech to text step, not the language model. Four things make Indian phone calls hard:

Phone audio. Ordinary voice calls carry a narrow band of sound, and mobile networks compress it further. Speech recognition trained on clean audio loses accuracy on calls.
Rare words. Personal names, surnames, local landmarks and your own product names appear rarely in general training data.
Code-mixing. "Mera order Thursday ko deliver hoga na?" mixes two languages in one breath. A model expecting only one language can trip over the switch.
Numbers spoken many ways. "Double nine", "nau nau", "ninety-nine" and digit by digit can all mean the same thing.

Background noise and speakerphone make each of these worse. Knowing the cause tells you which fix to reach for.

Choose speech recognition built for Indian calls

Speech to text models differ a lot on Indian telephony audio, and providers are now designing for it. Two examples from official documentation, read on 21 October 2026:

Sarvam says its Saaras speech to text models are optimised for 8 kHz telephony audio and support code-mixed speech, across 22 Indian languages plus English. Saaras v4 is listed as the default model, and adds keyterm prompting: up to 50 domain-specific names, places, brands or terms to bias recognition, on its REST and batch APIs.
OpenAI released GPT Transcribe and GPT Live Transcribe on 28 July 2026, which its changelog says support free-form transcription context, keyword hints and multiple expected input languages.

Note also that OpenAI has announced the shutdown of whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize on 26 February 2027. If any part of your setup relies on those, plan the move early. Our page on tracking provider changes explains how to stay ahead of such notices.

Six fixes that work on real calls

Read back and confirm. "That's Shraddha Kulkarni, pin code 4-1-1-0-3-8, correct?" One extra turn prevents a wrong booking or a wasted delivery trip.
Ask for one detail at a time. "Your name, address and preferred time?" in one question invites a long, error-prone answer.
Use the keypad for long numbers. Order IDs, pin codes and account references are often more reliable as keypad presses. See DTMF versus speech input.
Teach the agent your vocabulary. Put branch names, product names, doctors' names and nearby landmarks in the knowledge base, so the language model can recognise a near-miss and correct it.
Use what you already know. If the caller's number is in your CRM, greet them by the name on file and ask them to confirm, rather than asking them to spell it.
Spell out what matters in the script. Ask the caller to say the area and a landmark ("near Shivaji Park") rather than a full address.

Never ask a caller to read out an OTP, a full card number or a password, however good the recognition is.

Numbers, amounts and dates need special care

Detail	Common error	Better approach
Mobile number	Two digits swapped	Use the calling number, or read back in groups of 5
Pin code	One digit dropped	Keypad entry, then read back with the locality
Amount	₹15,000 heard as ₹50,000	Say the amount in words and figures, ask for a yes
Date	"Parso" or "next Friday" misread	Repeat the full date and day: "Friday, 30 October"
Order ID	Letters and digits confused	Keypad, or look up by phone number instead

The same rule applies on the speaking side. Indian voices from providers like Sarvam read numbers more naturally when your script writes them the way a person would say them. Our guide to writing voice scripts has examples.

Finding the words your agent keeps getting wrong

Guessing does not work here. The transcripts tell you exactly what is going wrong.

Once a week, read twenty random call transcripts and mark every misheard name, place or number.
Group them. If the same locality keeps failing, add it and its common spellings to the knowledge base.
Look for calls where the caller repeated themselves twice or more. Those are your worst experiences.
After a change of voice or speech provider, repeat the review, because accuracy shifts with the model.

Over a few weeks the list of problem words shrinks, and so does the number of callers asking for a person.

Improving recognition on Convo360

Convo360 runs on your own keys, so you choose the providers and see every call:

Pick the voice provider per agent from Sarvam, Smallest.ai, ElevenLabs or OpenAI, and test the same script on more than one.
Let the agent auto-detect the caller's language and switch mid-call, which helps with Hinglish and regional callers.
Add local names and terms to the agent's knowledge base.
Use the voice blocks, such as gather and DTMF menu, in a flow when you need a number entered on the keypad.
Set Background noise handling to match where callers are, under Call behaviour.
Review every transcript and AI summary in Calls.

See the voice docs for setup. AI voice calls are on the Pro plan at ₹2,999 per month, excluding 18% GST.

Example conversation

Haan, mera naam Venkatesh hai, aur delivery Gachibowli mein chahiye, DLF ke paas.

Thank you, Venkatesh. Just to confirm: delivery in Gachibowli, near DLF Cyber City. Is that right? I'll then share the slot options.

Build a voice agent that gets names right

Start free in minutes — no code, no credit card. Bring your own AI keys and pay no per-message markup.

Start 14-day free trial →See pricing

Frequently asked questions

Why does my voice bot not understand callers properly?

Usually because phone audio is low quality, the caller uses names or places the model rarely sees, they switch languages mid-sentence, or there is background noise. Choose speech recognition built for Indian telephony and add read-back confirmation for key details.

Which speech to text works best for Indian languages on calls?

There is no single answer for every business. Sarvam says its Saaras models are optimised for 8 kHz telephony audio and code-mixed speech across 22 Indian languages plus English. Test candidates on your own recorded call scenarios before choosing.

Should callers speak or type numbers on an AI call?

For short answers speech is fine. For long numbers such as order IDs, pin codes or account references, keypad entry is often more reliable, especially on noisy lines. Read the number back either way before acting on it.

Can I teach an AI voice agent local names and places?

Yes, in two ways. Add branch names, landmarks and product names to the agent's knowledge base so the language model can correct near-misses, and where your speech provider supports keyword hints, use them for your most important terms.

Are OpenAI's older transcription models being retired?

Yes. OpenAI's changelog of 26 August 2026 announced that whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize will shut down on 26 February 2027, with GPT Transcribe and GPT Live Transcribe as the replacements.