Text to speech and speech to text explained

Updated 2026-10-23 · 4 min read · By the Convo360 team

What TTS and STT actually do

Every AI phone call is really two translation jobs happening back to back. Speech to text (STT), also called automatic speech recognition, listens to the caller's voice and turns it into written words. Text to speech (TTS) does the reverse — it takes the AI's written reply and turns it into audio that sounds like a person talking.

Neither one understands what is being said. STT just transcribes; TTS just narrates. The actual understanding and reply-drafting happens in between, done by a language model reading the transcript and writing a response.

It helps to think of STT and TTS as the ears and mouth of the system, while the language model is the brain in between. Businesses often assume "the AI" is one single thing, but in a voice call it is really three separate services working together, each of which can be swapped, upgraded or fine-tuned on its own without touching the other two.

The full pipeline behind an AI voice call

A typical call runs through four stages, in a loop, for every turn of the conversation:

1. The caller speaks; the telephony line carries the audio to the platform.
2. STT transcribes that audio into text in real time.
3. The AI agent (an LLM) reads the transcript, along with the conversation history and its instructions, and drafts a reply.
4. TTS turns that reply into audio, which is streamed back down the line.

This repeats dozens of times in a single call. If any one stage is slow or inaccurate, the whole conversation feels off, even when the other three are working perfectly.

Why Indian accents and background noise trip up STT

Most STT models are trained mainly on American or British English speech. Indian English, regional accents, market or shop-floor noise, and switching mid-sentence between Hindi and English (common in real customer calls) all reduce accuracy if the model was not trained for it.

Mistranscription is not cosmetic — it changes what the AI thinks it heard. A caller saying "do sau" (two hundred) misheard as "do saal" (two years) can send the whole reply in the wrong direction. This is why choosing an STT provider with real Indian-language and Hinglish training data matters more than picking whichever one is cheapest.

Call environment adds another layer of difficulty. A customer calling from a two-wheeler in traffic, a factory floor, or a crowded shop counter produces audio very different from a quiet office recording used to train most models. Testing your chosen STT provider with a handful of real, noisy sample calls before launch, rather than trusting a clean demo, is the single most useful step to catch this early.

The latency budget: why milliseconds matter

In a normal human phone call, a reply gap longer than roughly one second starts to feel like an awkward pause. An AI voice agent has to fit STT, the LLM's thinking time, and TTS inside that same window, or the call feels robotic and people start talking over the bot.

This is why voice-first providers optimise for streaming: STT starts transcribing before the caller finishes speaking, the LLM starts drafting before the full transcript is ready, and TTS starts speaking the first words before the rest of the sentence is generated. It is a pipelining problem as much as an accuracy problem.

A common mistake is choosing the most accurate STT model and the most expressive TTS voice independently, without checking their combined round-trip time. A slightly less "polished" pairing that responds quickly often feels more natural in a live call than a technically superior pairing that leaves the caller waiting. This is one reason it is worth testing the full pipeline end to end, not each piece in isolation.

TTS and STT on Convo360

Convo360's AI voice agents use this same TTS/STT pipeline under the hood, with your own keys. You choose the STT and TTS providers per agent, and can pick Sarvam for India-native Hindi and regional-language accuracy, Smallest.ai or ElevenLabs for natural-sounding voices with voice cloning, or OpenAI voices for straightforward English support. Because it is BYOK, you pay each provider directly at their published rate — Convo360 adds no markup on the audio minutes or characters processed.

You can test different providers on the same script before committing, since accent handling genuinely varies by language and region.

Because the choice is made per agent rather than platform-wide, a business running both a Hindi-heavy support line and an English-heavy sales line can use two different provider pairings, each tuned to what that line actually needs, without any extra setup cost from Convo360 itself.

The speech-native alternative

A newer category of models processes audio directly, without converting to text as an intermediate step. These "speech-native" or "audio-to-audio" models can react faster and preserve tone of voice better, since nothing is lost in transcription. Gemini Live is one example already usable for voice calls. This approach is still maturing, but it points at where voice AI is heading — fewer hops, lower latency, and richer understanding of how something was said, not just what was said.

Example conversation

Frequently asked questions

Related reading

Available across India — see WhatsApp & AI calling automation by state and city.