Written by: Kathryn Collier, Data and AI Partner,
and Thomas Maas, Head of Client Solutions, Mantel
Key takeaways for business leaders:
- Speed without governance is high risk: Moving quickly without visibility, ownership, and feedback loops results in deferred rework, security vulnerabilities, compliance issues, and eroded trust rather than true acceleration.
- Establish proportionate foundations early: Organisations should implement practical safeguards, such as clear decision frameworks, robust access controls, pre- and post-release evaluation, and operational monitoring, tailored to the risk level of each AI use case.
- Make ambition and governance move together: Rather than waiting for an industry pause or perfect certainty, the smart approach for Australian enterprises is to build strong foundations early so they can experiment and scale AI initiatives safely.
Slow down, or build foundations?
Over the weekend, Anthropic CEO Dario Amodei argued that the AI industry must “slow the pace at which we improve the capabilities of AI models”, warning that the technology is moving faster than researchers can understand or control it. Sam Altman and Elon Musk agreed within hours.
For all the noise it kicked off, there is a serious lesson underpinning the argument.
In plain terms, Amodei’s worry is that AI has started helping to build the next generation of AI – “recursive self-improvement” – and that this has accelerated the frontier so much that the labs risk losing the time they need to test and safeguard each new model before the next one arrives. The ultimate fear is that humans will lose control over AI development, potentially leading to the emergence of a rogue AI.
His proposed response is to pace the race at the frontier: take the time to test and safeguard each model properly rather than sprint to the next.
That message is aimed squarely at the handful of companies building the most powerful models. However, there is a lesson in it for Australian enterprises too; just not necessarily the one it might first appear to be.
”The real risk isn't speed; it's ungoverned speed.
Kathryn CollierData and AI Partner, Mantel
What happens when organisations run ahead of their foundations
Moving quickly can create the appearance of progress while quietly increasing exposure. Common failure modes include:
- Sensitive information going where it should not. Staff may put confidential, personal or commercially sensitive information into tools that have not been approved, understood or configured appropriately. Once information has moved into an external system, it may be difficult to retrieve, delete or establish who can access it.
- Confidently wrong outputs entering real workflows. Without representative testing, evaluation and clear limits on automation, plausible but incorrect outputs can move into customer communications, operational decisions, analysis or code. The error may only become visible after it has created a cost or harm.
- Accountability becoming unclear. When a model, vendor, platform and several internal teams all contribute to an outcome, it can be difficult to answer basic questions: who approved the use case, who owns the decision, who monitors performance, and who acts when something goes wrong?
- Security risks expanding with every connection. Agents and AI-enabled applications often need access to documents, systems and actions. Poorly managed permissions, prompt injection, unsafe tool use or weak separation between environments can turn a useful assistant into a route to data exposure or unintended activity.
- Shadow AI becoming the operating model. If people cannot access safe, supported tools, they will often find their own. This creates inconsistent practices, fragmented purchasing, duplicated effort and limited visibility of where AI is being used.
- Compliance and legal obligations being discovered too late. Uses involving personal information, regulated decisions, intellectual property, records or third-party content may carry obligations that are difficult to meet after a system has already been deployed.
- Pilots accumulate without becoming reliable capability. Teams can produce impressive demonstrations that depend on manual work, a single expert or an unstable integration. Without standards for architecture, testing, monitoring and change management, each new pilot adds another exception to maintain.
- Costs and dependencies become hard to see. Usage-based model charges, duplicated platforms, rework and vendor lock-in can grow underneath a portfolio of experiments. The cost is not only the bill; it is also the effort required to operate and govern a system that was never designed for production.
- Trust eroding after avoidable failures. A small number of visible mistakes can make customers, employees and decision-makers less willing to use AI, including in areas where it could be valuable and appropriate.
None of these risks mean organisations should wait for perfect certainty. They do mean that speed without visibility, ownership and feedback loops is not acceleration. It is deferred rework and risk.
Strong foundations are what let organisations move faster
The practical alternative to a pause is not more process for its own sake. It is a small set of foundations that make responsible progress repeatable. These are:
- A clear decision framework. Define which uses are allowed, restricted or prohibited; what level of human review is required; and who is accountable for each material decision.
- Reliable information and access controls. Know what data is being used, where it can go, who can access it and how permissions are reviewed. Treat data quality and identity management as part of AI delivery, not a separate concern.
- Evaluation before and after release. Test for accuracy, bias, privacy, security, robustness and failure modes using realistic scenarios. Continue testing after release because models, data, prompts and surrounding systems change.
- Operational monitoring and intervention. Make it possible to see what a system is doing, detect drift or misuse, pause automation, implement kill switches to halt rogue AI behaviours, roll back changes and investigate incidents. This capability is critical for retaining organisational autonomy over AI.
- A path from experiment to production. Use consistent patterns for architecture, documentation, procurement, security review, change management and retirement. This prevents every new use case from becoming a bespoke project.
- Capability across the organisation. People need practical guidance on when to use AI, what not to put into it, how to check outputs and how to report problems. Governance works only when it is understandable and usable.
These foundations should be proportionate to the risk. A low-risk drafting assistant does not need the same controls as a system that influences eligibility, safety, financial outcomes or access to services. But every use should have a known purpose, an accountable owner and a way to detect when it is not working as intended.
The takeaway
Amodei is right that the frontier calls for caution. For everyone else, the smart move is not to wait. It is to make sure ambition and governance move together.
Organisations that build foundations early can experiment more freely because they know what is happening, who is responsible and how to intervene. Organisations that skip them may still move quickly for a while, but they will eventually pay through incidents, rework, stalled adoption or a loss of trust.
The question is not whether an organisation is moving fast on AI: it is whether its foundations can keep up.
If you’d like to talk to our experts about how to build strong foundations to scale your AI initiatives, get in touch.
See how we’re helping businesses scale with AI-first solutions
