Realtime Voice AI: Choosing Transcription, Conversation and Translation

A receptionist, a captioning tool, and a meeting-transcription app all use audio. They do not need the same architecture.

Start with the job the user expects: a spoken conversation, live captions, a transcript of a recording, translation, or narration from text. That choice determines the connection, events, and failures your application needs to handle.

Choose a conversation architecture

For new conversational applications, OpenAI's audio overview recommends starting with GPT-Live. It also describes the Realtime API's session and tool model.

A speech-to-speech system handles audio turns directly. A chained design converts speech to text, runs a text agent, and synthesizes the reply. The chained path gives you more explicit control over intermediate text; a direct audio path can support more natural turn-taking. Test both against your requirements.

Captions and recorded audio

Choose live transcription when users need captions from incoming audio rather than a spoken reply. OpenAI's current guide recommends gpt-live-transcribe, which returns incremental deltas and final transcripts.

The documented session flow requires client-side turn detection or explicit commits. Do not assume the same voice-activity settings work for every realtime transcription model.

Use file transcription for a completed recording or another bounded audio request. Streaming partial text from an uploaded file is different from processing an ongoing microphone stream.

The current file guide lists supported formats and a 25 MB limit. Check those details and any need for speaker labels before selecting a model. Some older transcription models are deprecated; check their migration deadlines and replacement features.

Translation and narration

Continuous speech translation has its own session model. Text-to-speech fits a different job: turning text into narration.

Changing only a model name may not migrate an integration. Session settings, transport, authentication, and event types can differ. Follow the current guide for the path you choose.

Connect the audio safely

OpenAI's overview points browser audio applications toward WebRTC and trusted server audio pipelines toward WebSockets. Keep long-lived API credentials on the server. When the chosen connection flow calls for short-lived credentials, issue them through a trusted backend.

For phone calls, design consent, caller identification, transfer, and fallback alongside the model integration.

Test with real-world conditions

Test what happens when a caller interrupts, pauses for a long time, has a poor connection, or moves to a human agent. Include background noise, names, phone numbers, and dates in the test audio.

Track time to the first useful response and incorrect or incomplete tasks. Fluent speech can still contain a wrong field. A transcript may also differ from what the audio model perceived, so do not treat it as infallible.

Before handling real calls, document consent, retention, access, escalation, and incident response. Use the worksheet below with representative audio and make a human fallback available. Report latency and task-success figures only after measuring your own stack.