OpenAI Realtime API
https://platform.openai.com/docs/guides/realtime
OpenAI's Realtime API for native speech-to-speech multimodal interaction with GPT-4o — launched Oct 2024 (DevDay). Skips the STT → LLM → TTS pipeline by passing speech directly to GPT-4o. The platform substrate that Vapi/Retell/Bland integrate as one of their LLM choices.
Active · Verified 2026-10-26 · fresh as of 2026-10-26 · 67 cited facts, 11 with a third-party source · Suggest a correction
At a glance
- Type
- Speech-to-speech LLM API (OpenAI Realtime) vendor fix
- Tier
- T1
- Section
- Voice agent platforms
- Latest release
- not applicable — not OSS fix
- License
- not applicable — not OSS fix
- GitHub
- openai/openai-realtime-agents — ~6.9k stars (MIT reference app on Agents SDK), last commit n/a; API itself closed source repo fix
- Funding
- OpenAI parent — $40B round Mar-2025 ($300B val) vendor fix
- Freshness
- 2026-10-26 fix
- Layer
- agent-infra repo fix
- Licensing
- commercial vendor fix
- SDK languages
- csharp, go, java, python, rest, ruby, typescript vendor fix
- Protocols
- mcp-client vendor fix
- Compliance
- ccpa, csa-star, gdpr, iso27001, iso27017, iso27018, iso27701, iso42001, soc2-type2, soc3 vendor fix
- Developer activity
- OpenRank 0.6 (12-mo avg 2.29) · 4 participants — 2026-09 third-party fix
When to use
Optimised for: Native speech-to-speech with GPT-4o; lowest-latency LLM voice tier
Anti-fit: Stateless platform — caller builds orchestration, memory, telephony adapter
Pros & cons
Pros
Native speech-to-speech (skips STT/TTS); OpenAI brand + price/perf; WebRTC support.
Cons
Audio token pricing is expensive at scale; stateless (caller builds memory/orchestration); OpenAI-only model choice.
Claims & capabilities
Launched DevDay Oct-2024; GPT-4o speech-to-speech native; WebSocket + WebRTC support
Technical surface
- API surface
- WebSocket + WebRTC + REST
- Backend storage
- Stateless session API; caller manages durability
- Deployment
- API (Cloud); no self-host
- Embedding model
- not applicable — not a memory product
- Multi-tenancy
- API multi-tenant
- MCP
- searched not found
- A2A
- searched not found
- OpenTelemetry
- searched not found
Compare OpenAI Realtime API with…
Alternatives to OpenAI Realtime API → live options ranked by shared design, licence, hosting and MCP support
Similar systems
Other voice agent platforms in the catalog, ranked by inbound references.
- Cartesia Sonic T1
Cartesia's Sonic TTS — state-space-model-based voice generation, lowest published TTS latency (~40ms). Cartesia raised $64M Series A Q1 2025 (Index + Lightspeed); founded by Mamba authors. Agent platform layer ('Cartesia Line') wraps Sonic with telephony + LLM.
- LiveKit Agents T1
LiveKit Agents — agent framework built on LiveKit's open-source WebRTC media server. LiveKit raised $45M Series B Sep-2024 ($300M val). Powers OpenAI Realtime API's WebRTC transport (LiveKit is the substrate OpenAI chose). Agents framework formally launched 2024.
- ElevenLabs Conversational AI (now ElevenAgents) T1
ElevenLabs launched Conversational AI Nov-2024 — STT + LLM + their own market-leading TTS in one pipeline. Combines vendor's voice cloning + multilingual TTS strengths with hosted call orchestration. ElevenLabs raised $180M Series C Jan-2025 at $3.3B val. As of 2026-10 ElevenLabs markets the platform as ElevenAgents.
- Pipecat (Daily) T1
Open-source Python framework for real-time voice + multimodal AI agents, maintained by Daily.co. Pipeline-style architecture (STT → context → LLM → TTS) wired over WebRTC. The OSS substrate that many of the voice platforms build on or compose with.
- Hume EVI T1
Hume's Empathic Voice Interface (EVI) — voice agent platform with vocal-affect understanding, using Hume's proprietary speech-prosody models. Hume will shut down the EVI and TTS APIs on 2026-11-13 (announced in its docs). Hume raised $50M Series B Mar-2024 (EQT).
- Speechmatics Flow T2
Speechmatics (UK; STT leader) launched Flow Q4 2024 — conversational voice-agent platform composed on Speechmatics STT + caller LLM. Speechmatics raised $62M Series B (2022); enterprise + media customers. UK / EU data-residency lead. As of 2026-10 the Flow pages and docs redirect to Agent STT, a speech-to-text API for voice agents (Linden model, from $0.30/hour) used via LiveKit, Pipecat or the API; Flow itself is no longer listed.
Sources (66 cited facts, 11 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Speech-to-speech LLM API (OpenAI Realtime) vendor
- What it does & what's distinct OpenAI's Realtime API for native speech-to-speech multimodal interaction with GPT-4o — l… vendor
- Claims / benefits Launched DevDay Oct-2024; GPT-4o speech-to-speech native; WebSocket + WebRTC support vendor
- Modalities voice + text (native multimodal) vendor
- Created 2024-10 vendor
- GitHub openai/openai-realtime-agents — ~6.9k stars (MIT reference app on Agents SDK), last comm… repo
- Mindshare Highest-impact OpenAI launch Oct 2024; many voice-AI platforms adopted it vendor
- Funding OpenAI parent — $40B round Mar-2025 ($300B val) vendor
- Customers / scale API customers (broad horizontal) vendor
- Pricing Audio token-based: $40/$80 per 1M input/output audio tokens (Oct-2024 launch) vendor
- Compliance SOC 2 Type II; HIPAA via OpenAI Enterprise vendor
- Data handling API zero-data-retention available; default 30-day abuse retention vendor
- Deployment API (Cloud); no self-host vendor
- HQ US (OpenAI, San Francisco) vendor
- Founders / pedigree OpenAI (Sam Altman, CEO) vendor
- Performance Sub-500ms turn latency typical; vendor claim ~320ms vendor
- Code/weights release Examples OSS — github.com/openai/openai-realtime-agents repo
- API surface WebSocket + WebRTC + REST vendor
- Latency p50/p99 ~320ms turn latency vendor claim vendor
- Backend storage Stateless session API; caller manages durability vendor
- Multi-tenancy API multi-tenant vendor
- Encryption TLS + AES-256 vendor
- SSO / RBAC via OpenAI Console / Enterprise vendor
- Versioning Model snapshots (gpt-4o-realtime-preview, etc.) vendor
- Namespace primitives per-API-key vendor
- Forgetting policy Zero retention available on API vendor
- Validated verticals horizontal — any voice agent vendor
- Time to running <1hr (API key + WebRTC integration) vendor
- Anti-fit Stateless platform — caller builds orchestration, memory, telephony adapter vendor
- Optimised for Native speech-to-speech with GPT-4o; lowest-latency LLM voice tier vendor
- Adjacent infrastructure Twilio (for PSTN); WebRTC clients; LiveKit Agents vendor
- Pros Native speech-to-speech (skips STT/TTS); OpenAI brand + price/perf; WebRTC support. vendor
- Cons Audio token pricing is expensive at scale; stateless (caller builds memory/orchestration… vendor
- Project & sources platform.openai.com/docs/guides/realtime; github.com/openai/openai-realtime-agents vendor
- obs-langsmith yes third-party
- obs-opentelemetry yes vendor
- obs-datadog yes third-party
- obs-helicone yes third-party
- obs-weave yes third-party
- obs-langfuse yes third-party
- obs-arize yes third-party
- obs-custom All major LLM-observability vendors auto-instrument OpenAI SDK calls vendor
- cost-token-budget yes vendor
- cost-prompt-caching yes vendor
- cost-semantic-caching no vendor
- cost-batching no vendor
- cost-model-routing no vendor
- cost-streaming-only yes vendor
- cost-observability-cost-attribution no vendor
- eval-langsmith-evals yes third-party
- eval-braintrust yes third-party
- eval-weights-and-biases-agent yes third-party
- eval-helicone-evals yes third-party
- eval-custom-test-harness no vendor
- eval-human-loop no vendor
- eval-production-traffic-replay no vendor
- commit-trajectory [{"ym":"2025-01","cum":2},{"ym":"2025-05","cum":31},{"ym":"2025-06","cum":37},{"ym":"202… repo
- architecture-deployment-locus cloud vendor
- licensing-bucket commercial vendor
- positioning-platform-vs-product platform vendor
- taxonomy-three-layer-rollup agent-infra repo
- developer-activity-opendigger {"activity":8.94,"as_of":"2026-09","openrank_12mo":2.29,"openrank_current":0.6,"particip… third-party
- compliance-certifications ccpa,csa-star,gdpr,iso27001,iso27017,iso27018,iso27701,iso42001,soc2-type2,soc3 vendor
- sdk-languages csharp,go,java,python,rest,ruby,typescript vendor
- protocol-support mcp-client vendor
- maturity ga vendor
Related systems
Referenced by (7)
- Helicone integrates with — Helicone docs have a OpenAI Realtime API integration page. (verified 2026-10-29)
- LangSmith integrates with — LangSmith docs have a tracing integration page for OpenAI Realtime API. (verified 2026-10-29)
- LiveKit Agents integrates with — LiveKit Agents docs have a OpenAI Realtime API model plugin page. (verified 2026-10-29)
- LlamaIndex (framework) integrates with — LlamaIndex has an official voice agent integration package llama-index-voice-agents-openai in its repo. (verified 2026-10-29)
- Mastra integrates with — Mastra repo ships the OpenAI Realtime API voice package. (verified 2026-10-29)
- Pipecat (Daily) integrates with — Pipecat docs have a OpenAI Realtime API service page. (verified 2026-10-29)
- Pydantic-AI integrates with — Pydantic AI realtime docs have an OpenAI page for gpt-realtime voice models (OpenAIRealtimeModel). (verified 2026-10-29)