Designing Realtime Speech Translation for a Multilingual Call

Translate speech or run an assistant?

For a multilingual call, a translation session streams what a person says into another language. An assistant session has a different job: it can answer, converse, and use tools. Decide which behavior the user expects before connecting audio.

OpenAI documents a dedicated Realtime translation endpoint using gpt-realtime-translate. Its translation session streams from incoming audio; it doesn't use the voice-agent response.create lifecycle.

Choose the transport for the audio source

Use WebRTC when a browser captures and plays media. The source audio goes through a media track, and translated audio is received as a remote track. The documented browser flow uses a short-lived client secret created by a trusted server.

A server that already receives raw media can use WebSockets. The provider guide describes base64-encoded 24 kHz PCM16 input for that path. Keep the standard API key on the server.

Keep speakers and output languages separate

When two people speak, keep their input tracks separate so each translation can be attributed to the correct speaker. The current guide recommends one session per output language. Make the active source and target language clear to participants.

Offer controls for the original audio, translated audio, subtitles, mute, and volume. A listener may need the original voice when a translation is uncertain or delayed.

Handle a continuous stream

The documented WebSocket translation flow sends audio continuously, including silence between phrases. Treat translation deltas as partial output; avoid presenting them as a final transcript before the corresponding completion signal. For WebSocket closure, follow session.close and wait for session.closed as described by the provider.

Do not add response.create to a translation session because a conversational sample used it. Test reconnects separately: a lost connection can leave the UI showing stale captions or an apparently active microphone.

Test accuracy where it matters

Make a small reference set with names, dates, numbers, currency, phone numbers, and the domain terms your callers use. Include mixed languages, accents, fast speech, and overlapping speakers. Compare the rendered source and translation, not just whether audio played.

• Measure first translated audio separately from end-of-utterance latency.
• Check whether a delayed subtitle stays attached to the right speaker.
• Include a reconnect and an intentionally unavailable translation session.
• Provide a way to switch back to original audio.

A small UI state model

Keep a clear state for each track so the interface can distinguish a muted source from an unavailable translation. This illustrative TypeScript type is application state, not an OpenAI event schema.

type TranslationTrack = {
  speakerId: string;
  targetLanguage: string;
  status: "connecting" | "active" | "reconnecting" | "stopped" | "error";
  partialText: string;
};

Walk through a two-person call

For a conversation between an English speaker and a Spanish speaker, decide where each participant hears translated speech and whether they can also hear the original. Keep each participant's source track and transcript associated with that participant. If you provide both directions, map each output to its intended listener and target language; don't play all translated streams to everyone by default.

The Realtime translation documentation recommends separate sessions for different output languages. Confirm the currently supported language codes and account limits before presenting choices in the UI. Display an explicit unavailable state for a language pair or connection that your implementation can't support.

A bilingual interface also needs a decision about text. Partial captions are useful during a turn, but a correction can arrive after the listener has read the first version. Use a visual distinction between partial and final text, preserve the speaker label, and avoid implying a partial caption is a certified record.

What to do when translation falls behind

A listener should be able to switch to original audio or subtitles when translated speech is delayed. Show that translation is reconnecting or unavailable instead of presenting an old transcript as live. If the connection returns, make it clear where the new session starts; a missing portion should not appear to have been translated successfully.

For an acceptance test, run a recorded exchange containing a name, a date, and a correction such as ‘Tuesday, sorry, Thursday.’ Ask a reviewer who understands both languages to compare the result with the source. Measure the first translated audio and the completed phrase separately, while also checking that the correction and speaker attribution are preserved.

This is especially important for domain-specific vocabulary. A smooth-sounding translation may still alter the term a listener needs. If the output will drive a consequential decision, build an appropriate human review or fallback into the workflow rather than assuming every phrase is correct.

Write down what the listener should hear if translation stops while the original speaker continues. The fallback might keep original audio available and show that translated audio has paused. Test that choice with the people who will use it, including anyone who depends on captions, and record the observed result. A reconnect indicator is useful only if the interface clears or updates it when the new connection actually succeeds.