Building GenAI Voice Agents: Implementation

Executive Summary

Most enterprise customer conversations still happen over the phone rather than in a web widget or a mobile app. If you want a voice agent doing real work, at some point it has to answer a 1-800 number. OpenAI's Realtime API now supports SIP (Session Initiation Protocol) natively, which means a phone call can route directly to the model with no media bridge or transcoding service in between, and the model responds in 200-300ms. That is close enough to human conversational cadence that the call feels like a conversation instead of an IVR tree.

That 200-300ms is also the number the rest of this guide is built around. Callers expect a response within about half a second, the model spends most of that budget itself, and whatever is left (roughly 100-200ms) is all your tools get. On the phone, retrieval latency, not speech, is what breaks the illusion. Our answer was to run the search index in-process and pre-load it while the greeting plays, and that pattern is where this guide ends up.

Getting there, we walk the whole path a call takes: SIP trunk configuration, the webhook that accepts the call, the WebSocket that carries session updates and tool calls, the low-latency retrieval pattern, and the production concerns that follow (multi-agent routing, compliance, observability, and scale).

The Case for Phone Integration

The case for the phone is short. It's the channel your customers already use: no app to download, no account to create, no microphone permissions to figure out. They dial a number and start talking, the same way they have for decades. That matters most for the people most likely to need help with insurance, healthcare, or government paperwork, who are often the least comfortable with digital interfaces. And demand doesn't keep business hours. The customer reviewing documents at 3 AM on a holiday weekend has always been the contact center's impossible staffing problem; an AI agent takes that call with the same attention as the peak-hours one, and capacity scales with demand instead of headcount.

There is an integration argument too. Because the agent connects over SIP, the same protocol the rest of the telephony ecosystem speaks, the contact center sees it as just another "agent" in the pool. It can take transfers from humans, hand back to them when a situation needs human judgment, and sit inside the quality-monitoring and compliance-recording workflows an enterprise has spent decades refining. You can adopt it incrementally without rebuilding the phone stack.