Introducing the Realtime API

Source: OpenAI Date: 2024-10-01 URL: https://openai.com/index/introducing-the-realtime-api

Summary

OpenAI launches the Realtime API — a WebSocket-based API enabling low-latency speech-to-speech interactions with GPT-4o, allowing developers to build voice AI applications with human-like conversational latency (sub-200ms response times). Unlike previous voice pipelines (STT → GPT → TTS), the Realtime API processes audio natively through GPT-4o’s multimodal architecture, preserving prosody, emotional tone, and conversational timing.

Implications

The voice AI platform thread. The Realtime API is the infrastructure that enables the voice AI application category to scale. Prior voice pipelines had noticeable latency and lost emotional information at each transcription/synthesis step; the native audio processing preserves what makes human conversation feel natural. This enables phone AI agents, voice-first applications, and real-time meeting assistants that weren’t previously viable at production quality.

Developer ecosystem impact. With the Realtime API, voice AI becomes a first-class developer primitive rather than a hack. Companies like Bland AI, Vapi, and others building voice agents now have access to GPT-4o’s native audio understanding. The Realtime API’s WebSocket architecture is lower friction than prior REST-based voice pipelines, accelerating the voice AI startup ecosystem and expanding the use cases for real-time AI interaction.