Australia’s government has woken up to the risks of AI. More ambition is needed

This week, Andrew Charlton, the federal assistant minister for science and technology, issued a stark warning about artificial intelligence (AI).

Speaking at the AI Safety Forum at the University of Sydney, he said powerful AI models “are already doing things their creators never intended: cheating, deceiving, going their own way”.

In response, the Australian government has established the AI Safety Institute under its National AI Plan. Charlton described it as “a national testing capability” and provided the most detailed public account yet of the work it will do.

But will it be enough to keep Australians safe from what could be one of the most powerful, rapidly advancing technologies the world has known?

The harms are already here

AI’s risks and harms are already affecting Australians.

There are the problems we are already seeing today: nudify apps, more sophisticated scams and deepfakes, voice cloning, chatbots that have isolated teens and encouraged harms, and heightened cybersecurity risks that are alarming our security agencies.

Beyond immediate and present harms, experts are sounding the alarm about what’s coming. This includes rapidly expanding capabilities and agentic AI that we don’t yet have controls for.

In a report published last week, the United Nations’ Independent International Scientific Panel on AI also raised concerns about global concentration of power, resources, AI capability, inequality, and the technology’s impact on how we think, reason and work.

A significant remit

The AI Safety Institute has three broad tasks, as Andrew Charlton explained at this week’s forum. First, analysing and testing new models. Second, supporting government regulators and agencies to respond to emerging AI capabilities, risks, harms and trends. Third, shaping safe AI development, deployment and international governance in Australia’s interests.

It is already doing work on multi-agent risks with the Gradient Institute).

You’ve probably already heard about AI agents that can book your table at a restaurant or plan your travel. The basic idea of an AI agent is an AI system that can plan and take action with limited supervision (for longer and longer periods too). When AI agents interact with each other without supervision, unpredictable risks can arise.

The AI Safety Institute is also working with the CSIRO on AI alignment – the science of trying to ensure that AI systems (and AI agents) act in ways that are aligned with their users’ values and goals. This matters because at this stage, we cannot fully predict how AI will act. There are plenty of experiments in which AI has deceived users or avoided human-imposed controls, even detecting (and underperforming) when it is being tested.

The head of the AI Safety Institute, philosopher and Royal Australian Air Force reservist Kate Conroy, also spoke at this week’s forum. She outlined plans to take on both immediate harms affecting Australians, with attention to those affecting the most marginalised and vulnerable, as well as frontier risks and harms.

So much to do

The creation of the institute is a critical step towards Australia taking the impacts of AI seriously.

But given the speed of AI development, and the range of issues that are emerging, it would be fair to ask whether its budget of A$29.4 million over four years is enough.

By contrast, the United Kingdom allocated around A$460 million in its 2025 spending review to the its AI Security Institute. Singapore’s AI Safety Institute has an annual budget of around A$11 million a year. Canada has allocated around A$50 million over five years to its AI Safety Institute.

All these budgets look insignificant next to industry spending. OpenAI alone sunk US$19 billion into research and development in 2025.

Proposals that the Australian institute’s budget should be significantly higher were strongly supported at this week’s AI Safety Forum.

The unequal impacts of AI and its huge potential to increase inequality, as highlighted by the UN’s new report, further highlight the fact there is much more work to do.

Two promising next steps would be to legislate a digital duty of care, to help address some AI-supercharged online harms, and to address digital exclusion to ensure all Australians can benefit from AI.

The Tech Policy Design Institute’s recent assessment also highlights Australia’s strengths but also key needs in areas such as securing computing power for research and public interest activities. This would ensure Australia can maintain its position in research, and that the public sector and civil society don’t fall behind the rest of the world.

Broadening the conversation

The conversation about AI, the opportunities it presents for society, and the impacts it will have needs to be much broader. We must consider a wider set of actions if we’re going to really address Australia’s low levels of trust in AI and keep people safe from the technology’s many potential harms.

With reports of a change in approach from the government, perhaps that’s coming.

As multiple speakers highlighted at the AI Safety Forum, Australia has a perhaps unique opportunity to make a difference. We have extraordinarily skilled AI safety researchers as well as people working on social impacts, as well as a societal and moral imperative to address the risks and opportunities of AI for all Australians.

Kimberlee Weatherall is a Chief Investigator with the ARC Centre of Excellence for Automated Decision Making & Society, funded by the Australian Research Council and a range of partner organisations, and Co-Director of the University of Sydney’s Centre for AI, Trust and Governance, which receives funding from a range of sources as declared on the Centre’s website, including IBM and Google. She is also a member of Australia’s Open Government Partnership Forum. Kimberlee is an honorary Fellow with the Gradient Institute, and was a member of the organising committee for the 2026 Australian AI Safety Forum.