By Jonathan Hardy | Principal AI Consultant, Mantel
Key takeaways for business leaders:
- Consumers already know what good voice AI feels like, and it’s making the gap between that and the average enterprise contact centre hard to ignore.
- Voice is a fundamentally harder engineering problem than text. Users expect responses in under 500 milliseconds; turn-taking, background noise, and telephony constraints add complexity that never shows up in a demo.
- For most enterprise deployments, a cascaded architecture (speech-to-text → LLM → text-to-speech) remains the proven choice. Speech-to-speech models are promising but not yet ready for compliance-sensitive environments.
- Guardrails and evaluation are non-negotiable. Containment rate, resolution rate, and cost per interaction are the metrics that matter.
- Organisations that get this right start with a constrained, well-defined use case rather than trying to overhaul entire contact centre functions at once.
Your customers are already talking and listening to AI. They’re playing audio briefings generated by NotebookLM and collaborating with Claude’s fully native voice mode. They’re telling ChatGPT to rewrite emails, out loud, hands-free, while making dinner. The interaction is natural, immediate, and increasingly becoming the baseline for how people expect to interact with AI. Then they call their bank, their insurer, their telco, and they hit a phone tree that hasn’t changed since 2005.
That dissonance is a market signal. Consumers now have a lived reference point for what a good voice interaction with AI feels like, and it’s making the gap between that experience and the average enterprise contact centre impossible to ignore. For Australian businesses, operating in a market where labour is scarce and expensive and customer expectations are rising, Voice AI is no longer an abstract technology trend. It’s a concrete opportunity to rethink how organisations and brands interface with their customers, and to shift from human-first support toward AI-led self-service in targeted areas.
“Voice AI will never be worse than it is today. Enterprises are already seeing material gains, both in contact centre and by layering voice on top of existing text-based agents. The tech capability is here, and it's already becoming table stakes.”
Jonathan HardyPrincipal AI Consultant | Mantel
Voice AI demand is real
If we look at the market backdrop, the global conversational AI market is projected to reach US$49.9 bn by 2031, growing at roughly 20% annually (Conversational AI Market Report). The contact centre AI segment alone is forecast to surpass US$12.9 billion by 2030 (P&S Intelligence). The investment is broad. Hyperscalers, including AWS, Google, and Microsoft are embedding Voice AI into their contact centre platforms. Frontier AI companies OpenAI and Anthropic are shipping voice-native capabilities. And specialist vendors such as ElevenLabs, Twilio, and Genesys are making large-scale bets on enterprise voice. Speech-to-speech models from OpenAI are crossing an enterprise-viability threshold. And on the demand side, clients are actively building Voice AI applications.
But “experimenting with Voice AI” and “being ready to deploy Voice AI” are two very different things. The technology has matured faster than most organisations’ understanding of what it takes to implement. The gap, between technology promise and readiness, is where the expensive mistakes happen – buying the wrong platform, underestimating delivery complexity, or building something that sounds impressive in a demo but is too fragile to launch to real customers where the experience is paramount.
In this blog, we map out the fundamentals of Voice AI, enabling you to ask the right questions and avoid common traps when it comes to leading your enterprise Voice AI initiatives.
Why voice AI is materially harder than text
Text-based AI and Voice-based AI are fundamentally different engineering challenges due to differing user expectations.
In a text chat, a two-second pause while the AI generates a response feels like thinking. On a phone call, a two-second silence feels like the line dropped. Users of voice applications expect responses within roughly 300-500 milliseconds (AssemblyAI). That’s the threshold where a conversation starts to feel natural, and hitting it consistently requires a level of engineering sophistication that text-based AI simply doesn’t demand.
A user’s tolerance towards latency is one key dimension. However, voice introduces turn-taking: knowing when someone has finished speaking and it’s safe to respond, without cutting them off or leaving an awkward pause. It introduces interruption handling, because callers will talk over the agent mid-sentence and the system needs to know what to do. It introduces the unpredictability of real-world audio, including background noise, accents, and the way people’s speech patterns change when they’re frustrated, distracted, or multitasking. And it introduces telephony, the reality that most enterprise voice interactions still happen over standard phone networks, which add latency, degrade audio quality, and constrain what’s technically achievable.
How Voice AI works, and why architecture choices matter
Every Voice AI system relies on four components working in concert: Speech-to-Text (the ears), a Large Language Model (the brain), Text-to-Speech (the voice), and an orchestration layer (the conductor) that manages turn-taking, interruptions, and conversational flow in real time. Get any one wrong and the experience breaks. These are the building blocks, but the more important question is how you wire them together.
The dominant approach for enterprise deployments today is the chained (or cascaded) architecture. Audio comes in, gets converted to text, the text goes to an LLM, and the response is converted back to audio. You can switch your STT provider without changing your LLM. You can upgrade your TTS engine without touching your orchestration. You get a text transcript at every step which is essential for observability, quality assurance and compliance. The trade-off is latency: the sequential handoff between three models typically adds seconds, though modern streaming architectures reduce latency significantly by processing data in overlapping chunks.
The emerging alternative is speech-to-speech (S2S). A single model processes audio in and produces audio and performs reasoning, eliminating the text intermediary. Response times drop significantly. The conversation flows naturally, and emotional tone is better preserved because no information is lost in the speech-to-text-to-speech conversion.
The excitement around S2S is warranted, but there are current limitations to be cognizant of for enterprise use. There’s no native text transcript, which creates real compliance challenges in regulated industries. Instruction following and reasoning are typically weaker than when using an LLM as part of a cascade architecture. It’s harder to add guardrails, plus evaluation and testing come with unique challenges as you’re dealing with audio rather than text. And critically, there are fewer S2S models deployed in Australia which introduces data sovereignty and latency considerations.
For most enterprise deployments today, a cascade architecture is the pragmatic and proven choice. S2S is worth experimenting with as it’s a promising emerging technology, but it’s not yet mature enough for the use cases where compliance, auditability, and reliability are non-negotiable (which is the case for any enterprise use case). Key to long-term viability is a modular architecture that supports both cascaded and speech-to-speech patterns, so components can be swapped as the space evolves. Investing in a stack that locks you into a single architecture or vendor is a decision you’ll revisit sooner than you’d like.
Demos? Easy.
Production? Hard.
Demos are deceptively easy with voice.
You can build a voice agent that handles a scripted conversation impressively in a matter of days. The latency is manageable when you’re on high-quality web audio. The edge cases don’t surface because the demo follows a happy path. Stakeholders see it, get excited, and assume production is a matter of weeks away.
However, production introduces everything from real-world latency challenges to callers who speak over the agent, background noise, unexpected questions, adversarial inputs, and the need to integrate with CRM systems, authentication workflows, and human escalation paths.
The gap between a demo and reliable solution is significant and requires engineering depth.
Production may also demand cross-channel continuity. A customer who starts on web chat and moves to voice shouldn’t have to repeat themselves. Context preservation, clean agent handoff, and consistent behaviour across channels are design requirements, and they need to be addressed in the architecture from the start.
The organisations that navigate this well tend to share two characteristics.
First, they start constrained. Rather than attempting to re-design entire contact centre functions, they begin with a well-defined use case that has clear boundaries. For example, inbound requests for product information, where the conversation structure is predictable and the stakes of getting it wrong are lower. Second, they layer voice on top of what already works. If an organisation already has a capable text-based agent handling customer inquiries, adding a voice front-end on top of that existing logic is a pragmatic extension into Voice AI. The voice layer handles the conversation; the existing agent handles the intelligence. It’s not always the right approach, and the technical architecture requires careful consideration, but it’s a pattern that avoids rebuilding from scratch.
Guardrails, evaluation, and build vs buy
A voice agent that provides medical advice it shouldn’t, leaks personal information through its transcription pipeline, or gets manipulated by an adversarial caller isn’t just a poor customer experience. It’s a brand and regulatory risk. Guardrails are a key design decision rather than a nice-to-have.
A layered approach is essential. System-level guardrails enforce non-negotiable rules: the agent will never provide medical, legal, or financial advice outside its scope. Supervisory guardrails monitor behaviour in real time, detecting topic drift, sentiment escalation, and patterns that suggest the interaction should route to a human. Adversarial testing is a must to stress-test the agent before launch against edge cases, prompt injection attempts, and deliberate manipulation. Voice-specific guardrails also include PII detection in transcription pipelines and content filtering. These layers are foundational for enterprise voice deployment.
Evaluation matters just as much as with text-based agents. Containment rate, the percentage of calls resolved without human handoff, is the headline metric. Customer satisfaction, average handle time, resolution rate, and cost per resolved interaction all contribute. A now established practice is using an LLM-as-a-judge approach: a separate AI model that automatically reviews and classifies calls against expected behaviour. Voice agents should be tested with executable scenarios, full transcript and trace analysis, audio playback review, and ideally CI-based regression loops rather than calling them a few times and hoping for the best. Evaluation, as with any generative AI solution, is essential.
And then there’s the build vs buy question. Custom Voice AI pipelines, using voice AI frameworks like LiveKit or PipeCat, offer deep control over orchestration and audio processing. But they demand engineering depth and a real appetite for experimentation. If that capability isn’t there, managed platforms like AWS Connect and Google Contact Centre AI offer a materially faster path to value. The build vs buy and configure decision should be driven by engineering capability.
Let’s keep the conversation going
Voice AI technologies are only getting more capable and the potential business value is clear, but the fundamentals – architecture choices, guardrails, technology selection, evaluations – need to be addressed head on. Getting them right early is the difference between deployments that scale and experiments that struggle to progress beyond the demo stage.
If you’re building Voice AI applications, or trying to figure out where to start, reach out to talk to Mantel’s Voice AI and Contact Centre experts.
