How we cut voice latency from 1.4s to 600ms
In late 2025 our p50 round-trip latency was 1.4s - too slow for natural turn-taking. Eight months later we hit 600ms. Here's what worked, ranked by impact.
The shortest path between “human said something” and “AI starts speaking back” runs through six stages: VAD, ASR, prompt build, LLM, TTS, audio out. In late 2025 our p50 looked like this:
1. Speculative LLM inference
The biggest win came from a technique we'd been skeptical of: speculative decoding. We run two LLMs in parallel - a smaller, faster draft model proposes tokens and the larger model accepts/rejects them. On our voice workload this dropped first-token from 520ms to 240ms.
“Speculative decoding only works because voice has tight semantic constraints - the draft model is right ~80% of the time.”
2. Pre-streaming TTS
We start streaming audio out as soon as we have the first phonetic unit - not the first sentence. Combined with a small buffer to handle TTS hesitations, this took TTS first-chunk from 320ms to 110ms.
3. Pinned regional inference
Latency to the LLM provider was murdering us - round trips of 80-140ms before the model even started thinking. We pinned inference into the same region as the call termination and gained ~70ms instantly.
function pickInferenceRegion( call: Call): Region {
// Co-locate model with media termination
const mediaRegion = call.media.region;
if (REGIONS.has( mediaRegion)) {
return mediaRegion;
}
return fallbackRegion(call);
}Wrapping up
We didn't find a silver bullet. Sub-second voice ended up being eight separate wins, in eight different layers of the stack. The lesson: every component is a latency budget; treat them all as user-facing.