ElevenLabs Realtime Voice Agents Trade Flexibility for Managed Simplicity

The new Eleven Realtime platform consolidates voice-agent capabilities into a managed offering, introducing Mini and Max models that handle speech recognition, reasoning, turn-taking, and voice generation together. Both rely on a cascaded architecture, though builders cannot select or independently tune the models underneath.

"With this launch, the company is effectively taking over the voice-agent harness, including turn detection, interruption handling, and orchestration." This approach shrinks the engineering burden of building and operating a voice-agent stack, yet constrains developers' freedom to debug, tune, or swap out individual pieces. Builders retain the ability to shape prompts, voices, tools, knowledge bases, and turn-taking behavior—including eagerness, timeouts, and interruption handling—via ElevenAgents.

A separate Custom mode pathway exists for teams unwilling to accept those constraints. This option permits developers to pick their own language and speech models and assemble the pipeline themselves. Realtime exchanges that degree of control for a turnkey solution. While builders keep influence over agent behavior, they forfeit authority over the models driving it.

Performance Results from Independent Testing

Artificial Analysis ran evaluations of both models across speech reasoning, conversational dynamics, and human preference. On Big Bench Audio, a collection of 1,000 spoken reasoning questions, Max achieved 98.8%, whereas Mini reached 75.9%.

Tool-use capability showed a starker separation. Max finished 87.1% of tasks successfully, while Mini completed 60.3%. "ElevenLabs recommends Max for agents that make frequent or complex tool calls." Teams constructing agents for reservations, account management, or other sequential workflows ought to evaluate this gap firsthand.

ElevenLabs positions Mini as the standard choice, arguing it suits the majority of scenarios. Mini also edged out Max slightly on Full-Duplex-Bench, which measures pauses, interruptions, and turn-taking, posting 95.8% against Max's 94.1%.

Latency numbers between the two remain fairly close. Max delivered 570 milliseconds to first audio on average, compared to 620 milliseconds for Mini. Artificial Analysis's June Speech-to-Speech Index documented a quicker 440 milliseconds for Deepslate Opal, though that earlier measurement does not constitute a direct comparison with Eleven Realtime.

Switching from OpenAI's Realtime API

ElevenLabs is pursuing organizations already operating OpenAI's Realtime API. According to the migration guide, Eleven Realtime uses the same protocol over the same WebSocket transport, meaning an application built on OpenAI's SDK can redirect to ElevenLabs without replacing SDKs—only the endpoint and model name change.

Switching carries technical costs. The endpoint operates on WebSocket exclusively, so a WebRTC application must change transports first. The SDK manages audio only via WebRTC. On WebSocket, builders must handle 24kHz PCM16 audio capture and playback themselves. A WebRTC app receives that capture and playback from the SDK currently, so migration demands constructing both.

Eleven Realtime operates separately from ElevenAgents. Phone and WhatsApp channels, knowledge bases, and monitoring all reside on the platform side. ElevenLabs advises teams needing those capabilities to construct on ElevenAgents initially, since a team starting with the bare API to move fast may need to rebuild later.

Mini versus Max Tradeoffs

Per ElevenAgents documentation, Mini and Max agents can route calls to each other, but neither can route calls to or from agents in Custom mode. Mini also lacks Zero Retention Mode and HIPAA configuration support. Max supports both in the US and EU, though it remains unavailable in India or Singapore data residency zones.

Both models use the standard ElevenAgents per-minute rate plus token charges. "Max costs more per token, but its stronger tool-use performance could make it cheaper for workloads where Mini requires more attempts to finish a task."

Testing Before Trusting

ElevenLabs manages the underlying models and orchestration, yet developers still require sufficient diagnostic data to identify what broke when a call fails. The extent to which Realtime exposes that information remains unclear.

The benchmarks look encouraging, but they do not demonstrate how Realtime behaves under real-world production conditions. Developers must verify whether performance endures and whether ElevenLabs supplies adequate information to trace failures.

ElevenLabs did not respond to The New Stack's request for comment by publication time.