Every voice demo I built before this year had the same tell. Ask the agent a question, and there's a beat of silence — a quarter-second, sometimes more — before it answers. That gap is the sound of a pipeline: speech-to-text transcribes your audio, a language model reads the transcript and writes a reply, then text-to-speech reads that reply back. Three models, three network hops, three chances to lose the tone of voice, the interruption, the "wait, no, the other order." It demos fine. It falls apart the moment a real customer talks over it.
On August 28, 2025, OpenAI moved its Realtime API out of beta and shipped gpt-realtime, its first general-availability speech-to-speech model. The headline is that a single model now handles audio in and audio out natively — no transcription round-trip in the middle. But the part that actually matters if you're shipping something is quieter: the API finally grew the plumbing that separates a good demo from a production system.
Why speech-to-speech is the real architectural shift
A chained pipeline throws away information at every seam. The transcriber flattens "I said fifty, not fifteen" into text and the tone that disambiguated it is gone. A native model keeps the audio as audio, reasoning over prosody and emphasis directly. That shows up in the benchmarks OpenAI published against the December 2024 preview model:
- Big Bench Audio (reasoning over spoken input): 82.8%, up from 65.6%.
- MultiChallenge (following multi-turn instructions): 30.5%, up from 20.6%.
- ComplexFuncBench (calling tools accurately from audio): 66.5%, up from 49.7%.
The MultiChallenge number looks low in absolute terms, and it's worth being honest about that — precise instruction-following over long spoken conversations is still hard. But the direction is what counts: roughly a 50% relative jump in the things that break voice agents in production, which are following the script and calling the right function with the right arguments.
There are two new voices, Cedar and Marin, available only through the Realtime API, and the model can switch languages mid-sentence and interpret system messages more faithfully than the preview. Those are nice. They aren't why I'd rebuild an agent on this.
The plumbing is the product
Four additions turn the Realtime API from a clever endpoint into something you can put in front of paying users.
SIP phone calling. The API speaks Session Initiation Protocol, so a voice agent connects to existing telephony — PBXs, desk phones, a support line — without a media-server sidecar translating between your app and the phone network. If you've ever wired Twilio to a websocket to a model, you know how much incidental code that erases.
Remote MCP servers. You attach a Model Context Protocol server by passing its URL into the session config, and the API handles the tool calls for you. Your inventory lookup, your CRM, your refund logic — the model reaches them through the same MCP server your other agents already use, instead of a bespoke function-calling shim per channel.
Image input. A caller can send a screenshot or a photo mid-conversation, and the agent grounds its answer in what it sees. "Which button do I press?" becomes answerable.
Reusable prompts. You save a bundle of developer messages, tools, variables, and example turns, then reference it across sessions — the same pattern the Responses API uses. Your agent's persona and guardrails live in one place, not copy-pasted into every session bootstrap.
Here's what wiring an MCP tool into a session looks like — a single session.update event over the websocket after you connect:
{
"type": "session.update",
"session": {
"type": "realtime",
"model": "gpt-realtime",
"instructions": "You are a support agent for an online store. Confirm the order number before issuing any refund.",
"output_modalities": ["audio"],
"audio": {
"output": { "voice": "cedar" }
},
"tools": [
{
"type": "mcp",
"name": "orders",
"server_url": "https://mcp.example.com/orders",
"server_label": "Order Management"
}
]
}
}
No client-side function dispatch table, no glue code to marshal arguments. The model decides to call a tool, the API routes it to your MCP server, and the result comes back into the conversation.
The realities that don't show up in a demo
Two things will shape your architecture more than the feature list.
The first is cost. Audio tokens are expensive: $32 per 1M audio input tokens and $64 per 1M audio output tokens, versus $4 and $16 for text. The saving grace is cached audio input at $0.40 per 1M tokens — so the way you manage context is now a line item. A long system prompt and tool schema replayed on every turn is real money at audio rates; caching and reusable prompts aren't conveniences, they're how you keep the bill sane. Overall, gpt-realtime runs about 20% cheaper than the gpt-4o-realtime-preview it replaces.
The second is that the model's context window is 32,000 tokens with a 4,096-token output cap. Audio consumes tokens fast. A long phone call will fill that window, so you need a strategy — summarize earlier turns, drop stale audio, or hand off — before the conversation runs out of room. Plan for it up front rather than discovering it when call number four goes quiet.
A voice agent's hardest moment isn't answering a question. It's being interrupted mid-answer, understanding why, and picking up the corrected thread without making the caller repeat themselves. Native speech-to-speech is what makes that recoverable instead of a reset.
The takeaway
If you have a voice agent stitched together from a transcriber, an LLM, and a TTS engine, the concrete move is to collapse the middle: run one gpt-realtime session, push your tools behind a remote MCP server so every channel shares the same logic, and budget context deliberately because audio tokens are priced to make you think about it. The model quality is good enough now that it's no longer the bottleneck — your interruption handling, your context strategy, and your telephony integration are. Build those well and the quarter-second gap disappears for good.
Sources: Introducing gpt-realtime and Realtime API updates for production voice agents (OpenAI) · gpt-realtime model reference (OpenAI API docs) · Realtime conversations guide (OpenAI API docs)