AI Voice Agent: Speech In, Speech Out, Low Latency

A voice agent is a speech loop: transcription turns the caller’s audio into text, your model decides what to say, and real-time text to speech answers out loud. Fish Audio provides the speech layer — speech to text, expressive streaming text to speech, cloning for a consistent brand voice, and the concurrency that keeps several conversations running at once. API access comes with the $99 Studio pack.

What makes a voice agent feel natural?

Three things separate a demo from something people will actually talk to:

• Latency. The gap between the caller finishing and the agent starting has to stay under about a second. Beyond that, people start talking over the agent.
• Interruption handling. Callers interrupt. The agent has to stop, listen, and answer the new question rather than finishing a paragraph nobody is waiting for.
• Voice consistency. A voice that changes cadence between turns sounds broken even when every word is correct.

Model quality gets the attention, but in practice the speech layer is where most of these three live.

Which parts does Fish Audio provide?

Four capabilities cover the speech side of the loop:

• Speech to text — inbound audio, with multispeaker separation and emotion tags, described in the speech to text guide.
• Real-time text to speech — streaming output so playback starts before the paragraph finishes; the text to speech guide covers emotion control.
• Voice cloning — a consistent brand or character voice from a short sample, with consent requirements covered in the cloning guide.
• Concurrency — the Studio pack carries the highest limit for simultaneous real-time streams.

Your dialogue logic, tools and retrieval layer stay in your stack. That is deliberate: the platform supplies speech, not the product decisions around it.

How do you set up the integration?

1. Buy the $99 Studio pack and create a key under Settings → API Keys — one for staging, one for production. The API guide walks through the key workflow.
2. Connect speech to text to your model, and your model to streaming text to speech.
3. Add interruption handling: when new audio arrives, cancel the current generation instead of queueing behind it.
4. Measure latency in three places — speech to text, the model turn, and time to first audio — so you know which stage to optimise.

Most teams discover that the model turn is not the slowest part; the speech-to-text step or a chatty prompt usually is.

What latency should you budget for?

Practical targets for a conversational product:

Aim for under a second in total, measured from the caller’s last syllable to the agent’s first. Measure on the network your real users have, not on office wifi — a demo that feels instant in the lab can feel broken on a mobile connection.

How do you pick a voice for an agent?

The right agent voice is the one that survives repetition. Four questions to settle it:

• Register. Support conversations favour calm and clear over dramatic; a voice that performs well in an ad may tire a listener on a support call.
• Pace. Read a real answer out loud in the audition — "your refund will arrive in three to five business days" tells you more than a demo paragraph.
• Consistency. Use one voice everywhere, and clone your own if the brand already has a spoken identity. Voices are cross-language, so the same agent can serve non-English callers.
• Rights. Cloning requires the speaker’s consent, and commercial use requires the Creator pack or above.

The voice library guide explains how to shortlist in minutes rather than hours.

What does it cost to run?

Budget from the words your agent speaks:

• Natural speech runs at roughly 150 words per minute.
• That is around 900 characters per minute of agent audio — about one credit.
• Transcription of the caller adds a comparable order of magnitude for typical turn-taking.

A pack-based model therefore scales linearly with call minutes, without a monthly platform fee:

Re-rendering is where the estimate moves: every prompt change that shortens or lengthens the reply changes the character count, so log characters per session from day one.

How should you test before launch?

Four rehearsals catch most production failures:

1. Interruption drills. Talk over the agent mid-sentence and confirm it stops and answers the new question.
2. Accent and language coverage. Test the languages your callers actually use, not just the one your team speaks.
3. Edge phrasing. Names, numbers, order IDs and dates are where speech breaks; test them explicitly.
4. Long sessions. Keep one conversation running for ten minutes and watch for latency creep and voice drift.

Record the sessions and keep the scripts. A regression suite of ten real conversations will tell you more after a change than any single demo call.

Where should you start?

Build the loop with one capability at a time:

1. Create a staging key from the Studio pack.
2. Stream text to speech first — it is the part callers notice.
3. Add speech to text and measure the full turn.
4. Only then add cloning, and only with the speaker’s consent.

The API guide covers keys, credits and concurrency; the rest of the loop belongs to your stack.

Frequently asked questions

What does Fish Audio provide for a voice agent?

The speech layer: speech to text for the inbound audio, real-time text to speech for the reply, voice cloning when the agent has to sound like your brand, and the streaming plus concurrency that conversational products need.

Do I need the API for a voice agent?

Yes. API access is included in the $99 Studio pack, which also carries the highest concurrency for real-time streams.

How much does a voice agent cost per minute of conversation?

One credit covers about 1,000 characters, and natural speech runs around 150 words per minute, so budget roughly one credit per minute of agent speech — transcription and generation together.

What latency should I aim for?

Under a second from the end of the caller’s sentence to the first syllable of the reply keeps a conversation feeling live. Measure it end to end on your own network before you commit to a stack.

Can the agent use our own voice?

Yes — clone a voice from a short sample with the rights holder’s consent, then reuse it across every session so the agent sounds consistent.