Leobit, a trusted .NET, AI, and web application development company, has released a comprehensive benchmark study comparing four real-time voice AI engines: OpenAI’s GPT Live, Google’s Gemini 3.8 Live, ElevenLabs Agents, and Azure GPT Realtime. This research is invaluable for businesses looking to deploy voice AI assistants that rely on retrieval-augmented generation (RAG) to answer questions from a company knowledge base. By evaluating response speed, accuracy, multilingual capabilities, and cost, Leobit provides a practical roadmap for making informed deployment decisions.
The study comes at a critical time, as these engines were only released weeks before the research began. Vendor demonstrations often fail to reveal how engines handle real-world challenges like noisy audio, language switching, or caller pauses. Leobit’s benchmark fills that gap, offering transparent, scenario-based testing that helps businesses avoid costly surprises in production.
What is the Leobit Voice AI Benchmark?
The Leobit Voice AI Benchmark is a detailed comparison of four leading real-time voice AI engines, conducted using identical instructions, data-retrieval functions, and prerecorded audio across 11 scripted scenarios. The tests also covered multilingual performance in German, Spanish, and French, as well as challenging audio conditions such as background noise and overlapping speech. This rigorous methodology ensures that the results are both reproducible and relevant for real-world applications.
Leobit initiated this research while selecting an engine for an internal voice assistant, making the benchmark a genuine reflection of the decision-making process that many businesses now face. The study’s findings highlight that published specifications do not predict real-world performance, underscoring the need for hands-on evaluation before committing to a solution.
What Stands Out
Response Speed and Cost Efficiency
Gemini 3.8 Live delivered the fastest substantive responses when knowledge-base retrieval was required, with a median of 1.43 seconds, and the lowest estimated cost per conversation—from $0.04 to $0.09 for a typical three-minute interaction. This combination makes it a strong candidate for high-volume voice assistants where cost per conversation is a critical factor.
Accuracy Under Challenging Conditions
GPT Live achieved the fastest initial audible response at a median of 0.79 seconds and the highest speech recognition accuracy overall. Under noisy conditions, both GPT Live and Gemini 3.8 Live performed exceptionally well, delivering fully correct responses in 83 and 80 out of 84 test runs, respectively. This reliability is crucial for phone and call-center environments where audio quality is unpredictable.
Multilingual and Knowledge-Retrieval Consistency
The study revealed that accurate speech recognition does not always lead to accurate responses. Azure GPT Realtime, for example, maintained strong transcription accuracy under noise but showed inconsistencies in language selection and knowledge-base retrieval, including generating inaccurate content in a French-language scenario. This finding emphasizes the importance of evaluating final answers in every supported language rather than relying solely on transcription metrics.
Who It’s For
The Leobit Voice AI Benchmark is designed for technology companies, startups, and enterprises that are evaluating voice AI engines for customer-facing or internal assistants. It is especially relevant for teams in call centers, customer support, and any business that relies on RAG to provide accurate, spoken answers from a knowledge base. The study also benefits developers and product managers who need to balance speed, accuracy, and cost when choosing a voice AI solution.
For project managers overseeing AI deployments, this benchmark offers a clear framework for comparing engines based on real-world scenarios rather than marketing claims. It helps teams identify potential pitfalls early, such as language switching in noisy environments or silence on caller pauses, which may not appear in controlled demos.
Pricing and Availability
The benchmark study itself is freely available on the Leobit blog, making it an accessible resource for any organization. While the engines tested have their own pricing models, the study provides estimated costs per conversation for Gemini 3.8 Live, ranging from $0.04 to $0.09, based on published pricing at the time of testing. Businesses are encouraged to evaluate their own usage patterns to get accurate cost projections.
Leobit’s consulting services are also available for companies that need deeper assistance with technology assessment, AI transformation, or custom voice AI development. These services are tailored to help clients select and deploy the right engine for their specific needs.
What Makes It Worth Considering
Leobit’s benchmark stands out because it goes beyond surface-level metrics to uncover real-world performance differences. The study’s use of identical test conditions across all engines ensures a fair comparison, and its focus on challenging audio and multilingual scenarios mirrors the conditions that voice assistants will face in production. This level of detail helps businesses avoid costly mistakes and choose an engine that aligns with their priorities.
Additionally, the benchmark is backed by Leobit’s extensive experience in AI and software development. The company’s team of seasoned architects and engineers brings a practical perspective to the evaluation, making the findings both credible and actionable. For any organization serious about deploying voice AI, this study is an essential starting point.
Verdict
Leobit’s Voice AI Benchmark is a must-read for businesses deploying real-time voice assistants, offering clear, scenario-based insights that help teams choose the right engine for their unique needs.
Frequently asked questions
GPT Live achieved the fastest initial audible response at a median of 0.79 seconds.
Gemini 3.8 Live had the lowest estimated cost, ranging from $0.04 to $0.09 per three-minute conversation.
All engines were tested with identical instructions, data-retrieval functions, and prerecorded audio across 11 scripted scenarios.
GPT Live and Gemini 3.8 Live delivered fully correct responses in 83 and 80 out of 84 test runs, respectively.
No, accurate speech recognition did not always translate into accurate responses, as seen with Azure GPT Realtime.
The right engine depends on the use case, and companies should evaluate candidates on their own scenarios before deployment.
The complete real-time voice AI benchmark is available on the Leobit blog.










