Key takeaways
Low-latency voice AI is not created by choosing a fast model and hoping for a natural conversation. It comes from measuring the full audio path, shortening endpoint detection, streaming instead of batching, reducing network hops, controlling tool latency, supporting interruption, and optimizing the slowest percentile rather than one impressive demo.
In this guide
The direct answer What latency means Why 300ms matters Voice architecture choices Build a latency budget Endpointing and VAD Streaming the response path Network and transport Tool-call latency Barge-in and interruption Telephony vs browser voice Observability Optimization playbook Production lessons FAQ Explore Agentic AI ↗The direct answer
To engineer fast voice AI, optimize the time between the user finishing a meaningful turn and hearing the first useful audio from the agent. That path includes endpoint detection, transport, model processing, optional tool calls, speech generation, buffering, and playback.
Trilops has optimized a production voice path to below 300 milliseconds in a specific measured configuration. We do not treat that number as a universal end-to-end guarantee. Network conditions, endpointing, telephony, tools, model choice, geography, and conversation state can all move the result. The useful engineering target is a measured latency budget with p50 and p95 behavior, not one best-case screenshot.
One word hides several clocks
Teams often report "latency" without defining where the timer starts and stops. That makes comparisons almost meaningless. A voice product can have fast model generation and still feel slow because endpoint detection waits too long. It can also produce a quick first audio chunk while taking much longer to complete a tool-backed answer.
This is controlled by voice activity detection, semantic turn detection, silence thresholds, audio quality, and conversation behavior.
Time to first model eventThis reflects transport, service load, context size, model choice, and whether the architecture is streaming.
Time to first audioThis is one of the most useful conversational metrics because people notice silence between turns immediately.
Full-turn latencyA response may start quickly while a booking, eligibility check, lookup, or record update continues behind it.
Interruption latencySlow cancellation makes an otherwise fast agent feel robotic because it keeps talking over the caller.
Tail latencyp95 and p99 behavior expose network variability, tool spikes, queueing, retries, and outlier payloads hidden by averages.
Conversation is turn taking, not benchmark theater
Human conversation depends on timing. Large pauses make people repeat themselves, assume the system failed, or begin another thought just as the agent starts speaking. Very aggressive endpointing creates the opposite failure: the system cuts users off before they finish.
The practical goal is therefore not "lowest milliseconds at any cost." It is fast, stable turn taking with correct interruption behavior and enough context to avoid premature responses. The optimal threshold differs by use case. A short appointment confirmation can be more aggressively tuned than a complex clinical intake answer where users pause naturally while recalling information.
Users do not care which service is waiting. They experience one gap between their speech and the agent's reply.
TrustA response that starts instantly but mishears identity, date, medication, or intent creates rework and erodes trust.
FlowWhen the agent is slow, callers repeat themselves, add filler, or interrupt, which increases downstream recognition and state complexity.
EconomicsTelephony minutes, realtime model usage, and agent occupancy can all rise when every turn contains avoidable waiting.
OpenAI's current Realtime documentation describes realtime sessions as the appropriate path for live audio that needs low latency, and supports realtime audio over WebRTC, WebSocket, and SIP. For browser and mobile clients, its WebRTC guide recommends WebRTC as the more robust client-side connection option.
Architecture determines how many serial waits you create
There are two common patterns: a native speech-to-speech realtime path and a chained speech-to-text, language-model, text-to-speech path. Neither is universally better. The right choice depends on latency, transcript requirements, deterministic processing, model flexibility, cost, compliance, and tool complexity.
| Factor | Native realtime speech-to-speech | Chained STT → LLM → TTS |
|---|---|---|
| Latency potential | Lower because the model can process and generate audio in one realtime session | More serial stages unless aggressively streamed and overlapped |
| Architecture | Fewer service boundaries | Separate transcription, reasoning, and synthesis components |
| Component control | Less ability to independently swap each speech stage | High control over STT, LLM, TTS, and domain-specific processing |
| Transcript pipeline | May require additional transcript handling depending on product needs | Transcript is a first-class intermediate artifact |
| Deterministic workflows | Works well when tool policy and output controls are designed around realtime events | Can be easier when each stage feeds strict application logic |
| Best fit | Natural, low-latency conversation and interruption-sensitive experiences | Workflows that need specialized STT/TTS, transcript processing, or independent vendor control |
Do not route every conversational turn through unrelated services, analytics jobs, or databases before audio can begin.
Business pathLogging, enrichment, summaries, analytics, and nonblocking writes can often happen asynchronously after the user hears the response.
Tool pathA confirmation phrase can sometimes start while a noncritical follow-up operation completes, but never speculate about a tool result that has not returned.
Fallback pathWhen a required API is slow, acknowledge the task, preserve state, and provide a safe wait or escalation path rather than leaving dead air.
Milliseconds need owners
A latency budget assigns a target to each stage of the critical path. The numbers below are an illustrative engineering budget, not a guarantee, benchmark, or claim about a specific vendor. The point is to make every wait visible and force tradeoffs before optimization becomes guesswork.
Important
Stages overlap in streaming systems, network conditions vary, and tool-backed turns may take much longer. Use this pattern to build a budget from your own traces. Report p50 and p95 separately for no-tool, read-tool, write-tool, browser, mobile, and telephony paths where those differ materially.
The cheapest latency win is often deciding sooner
Voice activity detection identifies when speech starts and stops. Endpointing decides whether a pause actually means the user finished a turn. These are related but not identical problems.
OpenAI's Realtime API supports voice activity detection for detecting speech start and stop events. Current documentation includes configurable server-side VAD approaches, including semantic turn detection that uses conversational context to estimate whether the user is finished.
Aggressive thresholds reduce dead air but can interrupt people who pause naturally between phrases.
Semantic endpointingA contextual turn detector can distinguish "I'm done" from a pause inside an unfinished sentence more intelligently than silence alone.
Noise floorBackground noise, echo, clipping, packet loss, and poor microphones can make speech detection less stable.
User typeOlder callers, multilingual users, clinical intake, and complex account numbers may involve longer pauses than simple FAQ conversations.
InterruptionThe system must detect the user starting again so it can stop playback and preserve the new turn.
TelemetryLatency dashboards should include user cutoffs, immediate corrections, repeated phrases, and endpoint wait time.
Do not tune endpointing only against milliseconds. A setting that saves 150 ms but frequently cuts off names, dates, medication doses, or appointment details is a regression.
Serial pipelines create serial waiting
Streaming reduces latency by allowing the next stage to begin before the previous stage is completely finished. Audio can flow continuously. Transcription can produce deltas. A model can start reasoning on partial context. Speech output can begin before the whole response is generated.
Very large audio or text chunks delay downstream work. Extremely small chunks create event, serialization, and networking overhead. Measure the real tradeoff.
ContextConversation history, retrieved knowledge, and system instructions should contain what the turn needs, not every fact the product has ever seen.
PreamblesFor longer tools, the agent can say it is checking availability before the result returns, as long as the preamble does not imply success.
Parallel workIf location hours and appointment availability are independent, they may be fetched concurrently rather than one after another.
Distance and transport are part of the product
Network latency is not something the model can optimize away. The audio path may cross a browser, mobile network, telephony carrier, media gateway, application server, model endpoint, tool provider, and back again.
| Path | Strength | Latency concern | Typical use |
|---|---|---|---|
| WebRTC | Designed for realtime media and browser/native communication | ICE negotiation, relay paths, jitter, device conditions | Web and mobile voice applications |
| WebSocket audio | Simple bidirectional application transport | Application-level buffering and TCP head-of-line behavior | Server-side realtime integrations |
| SIP | Direct fit for telephony infrastructure | Carrier routing, codecs, transcoding, PSTN path | Phone-based agents and contact centers |
| Telephony media stream | Connects live call audio to custom processing | Carrier plus WebSocket plus service hops | Programmable voice and legacy phone numbers |
Keep your realtime gateway, model connection, and critical tools geographically sensible. Avoid unnecessary cross-region round trips.
Connection reusePersistent realtime sessions remove repeated setup and authentication work from the conversational path.
CodecRepeated conversion between telephony and model audio formats adds compute, buffering, and quality loss.
JitterJitter buffers protect against variable packet arrival, but larger buffers increase playout delay.
Twilio Media Streams can send and receive live call audio over WebSockets for near-real-time processing. WebRTC is specifically designed for realtime audio and video transport. The practical choice depends on whether users are calling by phone or speaking through your own web or mobile application.
The slowest API often becomes the voice agent
A voice agent that only chats can feel fast. The real challenge appears when it must look up a patient, check availability, verify insurance, search a knowledge base, update a record, or send a message.
Does the result need to return before the agent can say anything useful?
→ 02Do not waste a network trip on missing or invalid identifiers.
→ 03Run independent lookups concurrently instead of serially.
→ 04Use timeouts, fallbacks, cached safe data, and escalation for slow dependencies.
Reference data and static configuration may be good candidates. Patient state or appointment availability may require a fresh check.
Write toolsRetries must not duplicate bookings, messages, tasks, or records.
PrefetchIf the user has already selected a location, the system can prepare relevant hours or service metadata before the next turn.
Preambles"Let me check that" can fill a legitimate wait. "You're booked" cannot be spoken until the booking succeeds.
TimeoutsDefine how long the agent waits before retrying, simplifying the request, or escalating.
MetricsOne slow third-party API should not hide inside a single "agent latency" chart.
Fast speech is not conversational if it cannot stop
Barge-in means the user can interrupt the agent naturally. The system detects new speech, stops or truncates the current audio response, preserves the correct conversation state, and processes the new turn.
"Actually, I need the afternoon, not the morning."
Measure time from user speech start to interruption event.
CancellationStop queued or buffered audio quickly enough that the agent does not talk over the user.
StateDo not preserve words the user never heard as if they were accepted conversational context.
ToolsDo not cancel an already committed external write merely because playback was interrupted unless the business workflow allows it.
EchoEnsure the agent's own playback does not falsely trigger user speech detection.
TestingInclude interruptions at the start, middle, and end of responses in every voice regression suite.
Phone calls add infrastructure you do not control
Browser and native applications can often connect directly to a realtime media service using WebRTC. A phone call may add PSTN routing, carrier infrastructure, codec conversion, telephony media streaming, and your application gateway before the model receives audio.
Measure by caller region and carrier where enough volume exists.
CodecTelephone audio has different bandwidth and encodingRecognition and endpoint behavior should be tested on real call audio, not laptop microphones only.
TranscodingEvery conversion can add buffering and quality lossMinimize unnecessary encode-decode cycles across the path.
Media gatewayThe gateway becomes part of your p95Trace receive, forward, model, return, and playback timestamps separately.
Call controlTransfers and DTMF create extra statesTest handoff, hold, disconnect, voicemail, and caller interruption behavior.
RealityTest on real networksOffice Wi-Fi and synthetic audio do not reproduce mobile calls, packet loss, noisy rooms, or Bluetooth headsets.
Optimization starts with one trace per turn
Every conversational turn should produce traceable timing events. The events should be safe for the data involved, but detailed enough to explain why one turn took 240 ms and another took 1.8 seconds.
speech started, meaningful speech ended, VAD decision, interruption detected, playback stopped.
Modelrequest/event sent, first model event, first output token, first audio delta, response complete.
Toolstool selected, call started, first byte, completed, timed out, retried, rejected.
Playbackfirst audio queued, first audio rendered, buffer depth, cancellation, underrun, completion.
Contexttransport, region, caller type, tool category, context size, audio duration, model, codec.
Outcometask success, correction, repeated user phrase, escalation, abandonment, and user interruption.
Illustrative trace only. It demonstrates instrumentation and should not be presented as a measured Trilops production sample unless replaced with a verified trace from your telemetry.
Optimize in descending order of measured pain
Add one shared trace ID and timestamps across capture, endpointing, model, tools, and playback before changing architecture.
Step 02Do not let slow external APIs distort the baseline conversational path.
Step 03Reduce unnecessary silence while tracking cutoffs, repeat phrases, and correction rate.
Step 04Use persistent low-latency media connections instead of record-upload-wait-response cycles for live conversation.
Step 05Process audio, transcription, reasoning, and speech output incrementally where the architecture supports it.
Step 06Remove irrelevant history, precompute stable instructions, and retrieve only what the current turn needs.
Step 07Run safe reads concurrently and move noncritical writes, analytics, and enrichment outside the first-audio path.
Step 08Reduce cross-region hops and unnecessary media gateways between caller, application, model, and tools.
Step 09Fix the slowest common path without increasing false endpoints, hallucinations, unauthorized actions, or user corrections.
Voice release scorecard
What 16+ production agents taught us
Once model latency dropped, endpointing, network distance, tool APIs, audio buffering, and application logic became visible. Optimizing only inference would have missed the actual user delay.
A no-tool FAQ turn, appointment lookup, write operation, transfer, and multilingual turn have different critical paths. Combining them into one average hid the engineering work.
We use the below-300ms proof point only for an optimized measured voice path, not as a blanket promise for every full conversation or external integration.
An agent that starts quickly but keeps talking after the caller interrupts still feels slow. Fast cancellation and correct conversational state are part of latency engineering.
When a schedule or record service slowed down, leaving silence was the worst interface. A truthful preamble, timeout, retry, or escalation preserved trust without inventing a result.
Some healthcare steps still need validation or human approval. We optimize around those controls rather than deleting them to win a latency benchmark.
Voice latency is a systems problem. The user hears the sum of every decision your architecture delayed.
Trilops production engineering principleFor the broader system architecture, read How to Build a Production-Ready AI Agent. For the healthcare voice workflow, see AI Voice Agents for Patient Intake. For release testing, see Evaluating LLMs for Healthcare.
Frequently asked questions
There is no universal number because the measurement boundary and workflow matter. For conversational quality, measure from meaningful end of user speech to first audible agent response, then track p50 and p95. Also measure cutoffs, barge-in, tool completion, and task success.
Specific optimized paths can reach below 300 milliseconds under defined conditions. That should not be interpreted as a guaranteed full-turn result across every network, telephony route, tool call, model, or geography. Publish the measurement boundary with the number.
Native realtime speech-to-speech removes some serial boundaries and can offer excellent latency. A carefully streamed chained architecture can still be appropriate when you need specialized transcription, synthesis, deterministic transcript processing, or independent vendor control.
It depends on the path. Common causes include slow endpoint detection, network distance, model response time, external tools, large context, audio buffering, telephony routing, and retries. Instrument every stage before optimizing.
A required tool can dominate the turn if it takes hundreds of milliseconds or seconds. Parallelize independent reads, prefetch safe data, use bounded timeouts, and speak truthful preambles where appropriate. Never announce a tool result before it succeeds.
The model may be only one part of the delay. Endpointing can wait too long, the client may buffer audio, the network can add round trips, or the agent may fail to stop quickly during interruption. User-perceived latency must be measured end to end.
Establish a healthy p50 baseline, then focus heavily on p95 because slow outliers damage trust and often reveal architectural bottlenecks. Keep quality metrics alongside latency so optimization does not increase cutoffs, wrong actions, or corrections.
Latency depends on architecture, network, model, region, transport, endpointing, tools, audio format, traffic, and measurement boundaries. Illustrative budgets and traces in this article are engineering examples, not universal performance guarantees.
Trilops builds production voice agents with streaming audio, measurable latency budgets, controlled tools, interruption handling, evaluation, observability, and healthcare-ready workflow safeguards.