Real-Time Text to Speech: Latency & Streaming Fish Audio
Build real-time text to speech that starts speaking quickly: streaming output, chunking, interruption handling, concurrency, latency budgets and credit planning.
Real-time text to speech streams audio while text is still being generated, so playback can start before the final sentence exists. The result is a conversation that responds in hundreds of milliseconds instead of going silent for several seconds. Fish Audio provides the speech layer: streaming text-to-speech output, speech-to-text for the inbound turn, voice cloning for consistency and concurrency for simultaneous sessions.
Why does time to first audio matter?
Users judge a voice interface by the silence after they stop talking. If the first syllable arrives after a noticeable pause, people repeat themselves, talk over the response or assume the connection failed. A strong average generation time does not fix that experience if the first audio packet is late.
Measure the path in three stages:
- End of speech to final transcript.
- Transcript to the model’s first sentence.
- First sentence to audible speech.
Most teams optimize the model first when the bottleneck is often the transcript or a slow text-to-speech handoff. Instrument each boundary separately before changing vendors.
What is a practical latency budget?
Under one second from the end of the user’s turn to the first reply syllable is a useful target for a conversational product. The budget is a constraint, not a guarantee: network distance, long prompts and cold connections can consume it quickly.
| Stage | Target | Common fix |
|---|---|---|
| Speech to transcript | 200–400 ms | Stream partial transcription and detect turn end earlier |
| Model to first sentence | 300–600 ms | Shorten prompts, cache context and avoid waiting for a long answer |
| Text to first audio | 200–400 ms | Stream audio, reuse connections and select a suitable model |
The ranges are engineering targets, not published service-level guarantees. Measure with the same network, microphone and audio player your users have.
How should streaming text be chunked?
Chunk by semantic boundary when possible: a sentence, a short turn or one clause that can stand alone. Very large chunks delay first audio. Very tiny chunks create choppy prosody and extra request overhead.
For conversational replies, a useful pattern is:
- Start with a short acknowledgement or first sentence.
- Continue with sentences that can be queued while the first chunk plays.
- Keep each chunk short enough to cancel when the user interrupts.
- Preserve a small overlap in context so names and tone remain consistent.
Do not split inside numbers, URLs, names or bracketed delivery instructions. Those fragments need to reach the model together.
How do you handle interruptions?
Interruption is a product rule before it is a transport feature. When new user audio arrives, the application should stop playback, cancel queued generation that has not started, and preserve enough context to answer the new turn. Audio already emitted cannot be taken back, so a short chunk size limits how much stale speech the user hears.
The voice agent guide covers turn detection and the full speech loop. The text to speech API guide covers the integration boundary without inventing undocumented endpoint fields.
Which voice sounds best in a live conversation?
Clarity beats theatrical range. A voice with stable consonants and a moderate pace is easier to follow when the listener may be driving, walking or listening through laptop speakers. Save extreme character reads for short performances or pre-generated assets.
Use delivery tags sparingly. A tag on every sentence makes the agent sound unstable because small differences in wording can change the performance. A consistent base voice plus occasional emphasis is usually more trustworthy.
How do you plan concurrency and cost?
Concurrency is the number of simultaneous streams your application can keep active. It affects queueing during peak load, not just the number of stored voices. Test with realistic turn lengths and background noise, then add headroom for retries.
Cost still follows generated characters. A 900-character reply uses about one credit, whether it arrives as one response or three streamed chunks. A one-minute spoken turn is commonly around 800–1,000 characters, so budget roughly one credit per minute of agent speech before retries.
The $99 Studio pack includes API access and the highest concurrency on this site. For pre-generated content, the standard text to speech guide is the simpler starting point.
What should you monitor in production?
Track time to first audio, total response duration, cancellation rate, stream failures and retry volume. Also sample finished conversations for pronunciation regressions, abrupt cutoffs and repeated acknowledgements. A fast agent that repeats the same line after every interruption still feels broken.
Log the script, voice and generation settings together, but keep user audio and personal data out of routine logs. A reproducible request is valuable for debugging; uncontrolled voice data is a privacy liability.
What should you test with real users?
Run scripted tests first, then listen to real conversations. Scripted tests prove that a stream opens, audio arrives and errors are handled. Real users reveal whether the agent starts too quickly, misses the end of a sentence or sounds unnatural when it interrupts itself.
Include noisy rooms, regional accents, speakerphones, barge-in and long pauses. Test a user who says nothing after the agent asks a question, and one who corrects themselves mid-sentence. Those cases put more pressure on turn detection than a clean two-person demo.
Measure both p50 and p95 time to first audio. The median can look excellent while a slow tail makes one in twenty callers wait several seconds. Alert on the tail, not only the average, and keep a text or pre-recorded fallback for requests that exceed the budget.
Record the first-audio timestamp at the player, not at the API response. A valid audio stream can still wait in a queue, buffer on a slow network or sit behind an autoplay restriction. The listener experiences the entire path, so the production metric should include every stage that can delay sound.
Frequently asked questions
What does real-time text to speech mean?
Real-time text to speech begins returning audio before the full text has been generated. Streaming playback reduces time to first audio, which keeps assistants and live interfaces from feeling silent.
What latency should a voice agent target?
Aim for under one second from the end of the caller’s turn to the first reply syllable. Measure speech-to-text, model reasoning and time to first audio separately so you know which stage is causing the delay.
Why should long text be chunked?
Smaller chunks let playback start sooner and make interruption handling easier. They also limit how much audio must be discarded when a user changes the conversation or a line needs to be regenerated.
Does streaming use fewer credits?
The credit cost follows generated speech volume, not the transport method. A shorter reply costs less because it has fewer characters; streaming changes latency, not the underlying price per character.
Can real-time text to speech sound expressive?
Yes. Delivery tags and voice selection still apply. The practical constraint is that generation must stay fast enough for the conversation, so short directed chunks often work better than one elaborate paragraph.