Build a Browser Voice Agent: WebRTC, Turn Taking, and Testing

Start with the conversation, not the transport

A voice assistant has to cope with more than a correct answer. The caller might interrupt it, pause while looking for an order number, or lose their connection halfway through a sentence. Decide what your application should do in those moments before choosing a model or writing a microphone button.

For a browser-based speech-to-speech agent, OpenAI's Realtime guide starts with the Agents SDK. Its higher-level session helpers are a practical starting point when you do not need to manage the underlying connection yourself. The lower-level WebRTC guide is useful when you need that control.

This walkthrough is an architecture guide, not a complete authentication or telephony implementation. Test against the official guide for the API and model you select.

Keep the standard API key on the server

OpenAI documents two browser WebRTC connection patterns. With the unified interface, the browser sends its session description to your server. Your server combines it with session configuration and authenticates the request to the Realtime API. The server is in the connection's critical path.

With the ephemeral-key flow, your server uses its standard API key to request a short-lived client secret. The browser then uses that secret to establish the connection. In either pattern, do not ship the standard API key in browser code.

Authenticate your own application endpoint and apply limits before issuing secrets. A temporary credential reduces exposure; it does not replace your product's user permissions or abuse controls.

• Request microphone access only when the user starts the experience.
• Keep private tool execution and its authorization checks on a trusted server.
• Make connection, error, and stopped states visible in the interface.

Use the right channel for audio and events

WebRTC carries microphone and generated audio through media tracks. OpenAI's WebRTC guide says the negotiated session description handles the audio format, so the WebRTC session should omit the audio format setting described for other transports.

Use the data channel for supported session commands and transcript or response events. Do not send microphone chunks as data-channel append events or expect audio delta events there. If your backend needs to observe the session or run private tools, OpenAI also documents a server-side sideband connection.

Release the microphone when the session ends

A useful UI has an explicit Stop control. It should close the session according to the selected API's graceful-close sequence, then release browser resources. The small helper below handles only local cleanup; it does not send the API's session-close event or wait for final usage. Add those steps from the current connection guide before invoking it.

// Browser resource cleanup only; call after your API's graceful-close flow.
export function releaseVoiceResources(
  peer: RTCPeerConnection,
  microphone: MediaStream,
  events?: RTCDataChannel,
): void {
  events?.close();
  peer.close();
  for (const track of microphone.getTracks()) {
    track.stop();
  }
}

Decide when a turn is finished

Voice activity detection, or VAD, identifies speech boundaries. For supported Realtime speech-to-speech sessions, server VAD and semantic VAD offer different ways to decide when the speaker has finished. Tune the experience against real conversation instead of assuming one pause length fits everyone.

Transcription sessions are model-specific. OpenAI's current guide says gpt-live-transcribe does not support server_vad or semantic_vad: omit turn detection or set it to null, detect the boundary on the client, and commit the buffered audio. Check the guide for your selected model before copying a session configuration from a different workflow.

Test the moments that usually get missed

Begin with a small set of recordings and live sessions that reflect the people who will use the product. Include the actual microphones, accents, background noise, and languages you expect. Do not store recordings or transcripts longer than your application's policies permit.

Measure more than whether the assistant sounded fluent. A booking assistant that repeats a wrong date smoothly has still failed the task. Record the outcome, time to first useful response, interruption behavior, and whether the user could reach a person when needed.

• The user speaks while the assistant is answering.
• A name, phone number, date, or currency is ambiguous.
• The network drops during a turn and the user reconnects.
• A tool is denied, times out, or needs a human decision.
• The user stops the session and microphone indicators turn off.

• Download the voice evaluation worksheet ↗