How to Measure Voice Agent Latency Without Hiding the Slow Parts

Measure the answer, not just the acknowledgement

A voice agent can respond quickly with ‘I’m checking’ and still leave the caller waiting for the information they need. Record the acknowledgement and the first useful answer as separate moments.

Start with a repeatable scenario, such as a caller asking for an available appointment. Preserve the recording, configuration, and expected outcome so you can compare a change against the same task.

Mark the stages you control

Capture timestamps for connection setup, turn end, backend request start, tool start and end, useful result, audio arrival, and client playback when your architecture exposes those events. OpenAI's voice-agent guide describes additional stages for GPT-Live delegation.

Use a monotonic clock within a process for durations. When events cross machines, don't subtract wall-clock timestamps as if their clocks were automatically synchronized. Record each span at the service that observes both ends or use tracing designed for distributed systems.

Calculate a duration honestly

The small helper below measures one local span. It doesn't measure network time elsewhere or tell you whether the answer was correct.

const started = performance.now();
await performToolCall(); // Your application function.
const elapsedMs = performance.now() - started;
console.log({ stage: "tool", elapsedMs });

Compare distributions, not one demo

Gather repeated runs and report the median and a slow-tail measure such as p95. Keep the caller, recording, backend model, prompt, tools, transport, and audio cadence consistent. Change one factor at a time.

Track task completion beside latency. A faster assistant that books the wrong date is not an improvement for the caller.

Investigate where waiting occurs

If connection setup is slow, inspect the credential and network path. If the backend waits on tools, measure each tool and its timeout separately. If audio arrives quickly but playback starts late, inspect buffering and the client audio path.

Streaming, caching, and smaller tool sets can help particular bottlenecks, but their benefit must be measured in your own workflow. Keep a human fallback when an operation times out.

Plan cost alongside latency

For GPT-Live, the provider documents voice session duration and backend work as separate cost components. An active session can include silence and backend wait time. Muting a microphone does not close the session; follow the documented close flow and collect final usage.

Record the workload, usage, and current pricing when you calculate cost. Do not reuse a sample rate as a quote for your deployment.

Create a timeline for one call

Imagine a caller asks for an appointment and the assistant checks a scheduling service. Your timeline might include user turn end, scheduling request start, scheduling response, first answer audio received, and audible playback. Separating those points tells you whether the wait was caused by the scheduler, model processing, connection, or playback path.

Keep the identifiers needed to join those observations. If the scheduling service runs on another server, have that service measure its own request span or use a tracing system that handles clock differences. A browser's performance.now() value is useful within that browser; it isn't a shared timestamp that can be directly compared with another machine's monotonic clock.

Record interruptions and errors as outcomes, not as unusually short successful calls. A disconnected session may have a small duration while providing no useful answer. Define how incomplete runs are classified before computing percentiles.

Use a baseline to decide what to optimize

Run the same scenarios several times under a documented environment. Capture the model and prompt version, transport, audio source, tools, and result. Then change one factor, such as a slow tool timeout or response buffering policy, and repeat the set. Compare median and p95 as well as task success and interruption handling.

A useful report might say that a particular test scenario spent most of its observed time in the scheduling tool. It should not promise the same improvement to every user or omit failed runs. Keep the raw measurement definitions with the numbers so another teammate can reproduce the calculation.

After the conversation ends, follow the documented session-close process and collect final usage where available. Measure total cost per completed task separately from user-perceived latency: lower latency may increase usage, while a cheap interaction that fails often can create support work.

Include runs where the caller interrupts, the network reconnects, and a tool takes longer than its timeout. Record them separately from normal completions so the fastest and slowest paths aren't blended into a misleading average. If a change reduces first audio time but makes the final answer less accurate, the test report should show both results. Review the same synthetic scenario before and after the change and keep the original trace for comparison.