Inside OpenAI’s six-month rebuild of GPT-Live’s realtime voice stack
OpenAI rebuilt its realtime voice architecture in six months to support GPT-Live, its third-generation voice system. The full-duplex model can listen and speak simultaneously, removing the separate turn detector that earlier systems used to decide when a user had finished.
The resulting platform streams incoming audio directly into the voice model and returns speech while deeper reasoning and tool calls run on an asynchronous path. GPT-Live can consult frontier models such as GPT-5.5 without stopping the conversation, and the same foundation powers ChatGPT Voice features including computer control and agent coordination in the desktop app.
Keeping audio on the critical path
Earlier cascaded systems ran speech-to-text, a language model and text-to-speech in sequence. Speech-to-speech models preserved more tone and pacing, but still waited for a turn detector before inference began. GPT-Live instead maintains a continuous media loop, leaving persistence, tools and frontier-model delegation outside the live path.
The media frontend and inference logic were rewritten in Go, replacing a Python asyncio implementation. OpenAI says the new system’s p95 frame-delivery performance matches the previous system’s p50. WebRTC handles low-latency transport and can compensate for packet loss, clock drift and connection changes by adjusting audio playback.
This architecture extends GPT-Live’s role in ChatGPT voice interaction with implementation details on how continuous inference, transport and delegation work together while preserving conversational responsiveness at ChatGPT scale.
Managing state without audible pauses
Long voice sessions continuously accumulate context while model instances change with demand. OpenAI’s handoff mechanism warms a replacement instance, prefills it with the current context, runs both instances in parallel and switches only when the replacement is ready.
The same process supports context compaction. Rather than pause while compacting history and rebuilding the key-value cache, the system prepares a replacement in parallel as the original instance continues the conversation. It then cuts over without interrupting media.
Delegation follows a similar principle. An application server creates and prefills a GPT-5.5 inference session when the voice session begins, then retains session affinity and uses prompt caching. Routing, prompt processing, inference, tool calls and model-tool round trips all remain part of the latency budget.
Reducing startup and validating real workloads
OpenAI also developed the WebRTC Abridged Roundtrip Protocol, or WARP, to reduce media and data startup from six network round trips to one. Its backward-compatible changes include carrying the DTLS handshake over ICE, using DTLS 1.3, and pre-negotiating SCTP and data channels. The specifications are moving through the IETF TSVWG, with support already added to libwebrtc and Pion.
Instant Connect removes the SDP signaling exchange from the critical path by negotiating parameters ahead of time without reserving server capacity. Combined with WARP, it lets a client begin a session with one UDP packet; standard signaling remains available as a fallback if the prepared parameters are invalid or stale.
Before launch, OpenAI shadowed a gradually increasing share of production ChatGPT Voice sessions while Advanced Voice Mode continued serving users. The tests showed that capacity depends on CPU stream handlers, queues and networks as well as GPU throughput. They also exposed regional latency, long-session memory pressure, reconnection issues and shutdown races.
For businesses building voice services, the practical implication is to isolate media from application logic and assess responsiveness across the entire session lifecycle. Concurrent-session capacity, regional routing, state recovery, tool latency and per-frame delivery must be tested together rather than treating model inference speed as the sole performance measure.