Breakfast Wrap: Questions swirl around Australia's rogue AI intrusion

0:00A pig gets shot and this nearly starts a war. A pastry shop gets trashed. War. A football match turns ugly. Also war? Hey there. I'm Mark Vannell. On the No One Suriccoming podcast, we are taking on history's most absurd conflicts. Four episodes, four wars and we called it this stupid war. Research No One Suriccoming podcast on ABC Listen, or wherever you get your podcasts. From Gadigal Country, I'm Nick Green. And today on the Breakfast Wrap, we go deeper into the hack on the Australian Government's online data by Rogue Artificial Intelligence agents. We put questions on the minds of radio national breakfast listeners to experts for their analysis on what it all means. AI tools should not have done this, but also the infrastructure that supports the data

1:05is vulnerable, they're ignoring instructions about the law and so on. We also get the latest from the Albanese government as it investigates how the intrusion occurred and why it took so long for Australians to be told about what had happened. Now is it that AI moves so fast that we learn about what it's doing so slowly. Coming up, we'll get analysis two of Donald Trump's historic meeting with China's Xi Jinping. I always hear that the overall relationship now is kind of stuck. All that and more, from the Radio National Breakfast Team, this is the Breakfast Wrap. Artificial Intelligence. It's been advancing at such lightning pace that those engaged in its development have often marveled with a little exasperation at times that what they learn about the technology's new capacities today is all too frequently obsolete by tomorrow. It's pretty much moving faster than human brains can comprehend.

2:06In fact, it's worth contemplating the contrast between how fast AI has learned to escape its constraints and launch cyber attacks on sites like Australia's Medicare Data Portal, as we learned only yesterday, and how long it took months in fact for its designers to realise they had a problem they needed to confess to the world. Australia's National Cybersecurity Coordinator, Lieutenant General Michelle McGinnis, says everyone needs to do better. We need to be informed sooner of this incident. The systems that are set up have been designed for humans who find vulnerabilities to notify government. This is a very fast moving space. It experts say the incident raises urgent questions about the boundaries of AI development and what protections are needed against overreach in Australia. For more on that and to try to unpack this huge issue, Radio National Breakfast invited listeners today to send in their questions about AI-going rogue. And to answer those questions, Sally Sarah was joined by John Payne, the chair of the

3:07Independent Digital Rights Advocacy Group, Electronic Frontier's Australia. She spoke to Bill Simpson-Young, CEO of the Gradient Institute. That's an independent not-for-profit AI research institute. And she started by asking him for a response to one listener's question about whether the owners of the technology should be prosecuted. Yeah, so my understanding is, and I'm not a lawyer, but it's still a bit unknown exactly where this fits. It is thought by some lawyers that it does fit under product safety laws, but it's been well-creditsised that product safety laws are not really tied enough to deal with this type of situation. So I think it's still being determined whether there is something that can be done. My view is that we should be improving product safety laws so it's clear that they do apply to AI systems like this. Another question says, why is the open AI hack being treated as an AI emergency rather than a security vulnerability in the platform that was hacked? What do you think?

4:07Certainly, it's being characterised as a hack, and that's probably just an experience in term to use in the media. What it is is a tool that is gone and doing research. The issue is it's too cited, the AI tools should not have done this, but also the infrastructure that supports the data in Medicare is vulnerable. There are requirements, for example, under the Australian systems directory guidelines to maintain a particular maturity level in relation to the government's own Commonwealth Subsecurity Poster Report, and that was placed in 2025. And I've found that on a 22% of Commonwealth entities had reached that maturity level. So if the Medicaid portal was not at the required maturity level, so Service Australia itself was failing to meet the government's own mandatory policy baseline even before the open

5:08agent arrived from, sorry, the agent arrived from open AI. And that begs the question, if Services Australia's basic security is at such a low level, doesn't put the government's entire digital ID strategy at risk? Another listener question I was going to ask you exactly the same thing. If AI is trained by hoovering up information, including laws, why are can't it be instructed not to break these laws? AI's are being trained to be very persistent so that they can attempt to solve very difficult tasks to solve. We've seen recently that AI has been used to solve major mathematical problems that have ever been solved before. But that's not just single agents, it's actually thousands of agents. There was 10,000 agents that we used together working together to solve that particular problem. What you're getting now in situations like this and in the previous open AI agent and take a plug-in phase, it's not just one agent going out and doing this, it's usually a

6:08situation where companies that are frontier like open AI are training large numbers of agents at the same time, those, sorry, train large numbers of models at the same time, those models are ending up being able to find each other and discuss how to hack into websites, how to cheat and so on. They might have a fairly innocuous long-term goal and that goal might be, as in this case, understand spending on skincare products in Australia. But when they're trying to solve that final goal, they will form some intermediate or called instrumental goals. Those could involve just going off and finding, getting access to resources, getting access to information that needs to solve the final goal. Just of the level of persistence they've been given, they're just going off there and doing that. They're ignoring any sort of instructions about the law and so on and just acting very persistently toward that final goal even though it might be fairly innocuous. Another question. We've heard the government trying to push to get some of the frontier AI models here in

7:10Australia. Would that create greater danger of these kinds of incidents or the borders not really matter when it comes to AI, even if these models are being trained and run from the US, they can still, the borders won't affect them in investigating and or crashing into places they shouldn't hear in Australia. That's a really interesting question because it goes broader than the AI systems and cells. It also goes to the infrastructure that's required to support and house AI tools and access to AI services. That's a really big question because the government around federal level and civil states are running inquiries into how or under what circumstances AI infrastructure should be allowed in Australia. There's a question around it also around our own data sovereignty. That's a question that someone doesn't appear to grasp but around government and it's

8:10a very, very important thing. I just add to that, I think it's really important to, if the models are trained in Australia, the Australian government, Australian people can have much more oversight of them and we're much more likely to be able to set the rules that have all the product safety rules and have much more control of them and try to help, help make sure that the companies are building the models safely and building models that end up being safe. Doing that when the models are being trained overseas is near impossible. We do need the data centers. They need to be done with the right level of water use and energy use and so on in the right places but they should be in Australia. It just gives us much more control and people will end up in a much better place. Do the agents tell the truth to their handlers when they're asked how they got this information or do they lie if they know they're not supposed to be breaking the law? Yeah, they lie. There's lots of evidence that they will be quite deceitful and if they'll be heading towards their long-term goal and in order to achieve that long-term goal if it means lying

9:13that will and a lot of it goes into trying to make the models not live but evidence keeps coming out that they will be quite deceitful. Bill Simpson Young is the CEO of the Gradient Institute and Independent Not-for-Profit AI Research Institute and John Payne who we heard earlier is the chair of the independent digital rights group Electronic Frontiers Australia. Today Australian Prime Minister Anthony Albanese is urging nations to ban together and cooperate to confront the challenge of runaway AI with the same urgency as climate change and the unraveling of the rules-based international order. But questions remain about why Australians were not told sooner about health records being hacked by rogue AI agents. Today Sally Sarah put those to the Assistant Minister for Science, Technology and the Digital Economy Andrew Chalton. Can we clarify the timeline here? The government was notified about the hack two weeks ago. The Prime Minister was told on the weekend why was the Australian public only informed yesterday?

10:19Well, it took a considerable amount of time for open AI to provide this information to the Australian government. We were first notified on the 10th of September, even though this incident occurred in June, five days later on the 15th of September, services in Australia notified the Australian signals direct grip, which is the agency responsible for assisting in this type of incident. The responsibility of the Australian signals direct is to determine the nature of the breach. It was important for us to be able to determine the type of information that was accessed in particular to be able to inform Australians about whether or not any personal data had been breached. In this case, we'd have determined that it has not been breached. Then the Prime Minister then spoke to the CEO of Open AI and announced this to the Australian public. The New South Wales Premier was only notified about potential breaches for a state government website on Wednesday. Why was he only told then and what was the threshold for going public when you did?

11:21Well, we wanted to make sure that we provided this information to the Australian public as quickly as possible in relation to the engagement between the New South Wales government and Open AI. I'm not aware of the nature of the communication between those two organisations, but from the Commonwealth's perspective, we wanted to provide this information to Australians as quickly as possible once the facts had been established. Why did it take five days for services Australia to notify ASD and start escalating this issue? Well, the nature of the reporting by Open AI was completely inadequate. In this type of situation, you would expect, for an incident which is as serious as this, you would expect that disclosure to be delivered in a very serious way. It was not delivered in a very serious way. It was delivered via an email to a public inbox. One of the reporting requirements that we expect going forwards is that the nature of the disclosure is consistent with the seriousness of the incident and that means it being reported

12:28at a senior level and with full information and that will assist in the rapid escalation of these types of incidents. Is the government planning to take any legal action against Open AI? Well, the government has said that we will take advice on a referral of this incident to the AFP. This is clearly an important cyber breach and it's right that it will be investigated. We're also looking through a rapid review at whether there are changes to the Australian law that will be required to recognise this type of breach and it's propensity going forward. If it wasn't AI getting into this portal, if it was a human individual, would that individual be liable for legal action or charges for making unauthorised access? Look, you are right to point out this important legal question and AI agent is not a legal person and in our law liability for this type of incident always sits with a person or a company. In this situation where the breach was conducted by an AI agent, the liability has to be traced

13:34back to the intent of a person or a company that created or directed the agent. So these are important legal questions and that's why we're taking advice on them in order to either make a referral or make changes to the law that may be required. How quickly does the government want this rapid review completed? Well, we want it done quickly. There's a specific time frame but we want it done quickly because we are very keen to make sure that it informs the legislation that the Prime Minister has said will be out before the end of the year. That legislation is the Australian AI standard that the Prime Minister announced a few months ago. One of the components of that Australian AI standard was around precisely these issues, safety, transparency and disclosure and clearly this incident and the review that is being undertaken will inform the landing point of those standards in this area. Assistant Minister for Science, Technology and the Digital Economy, Andrew Chalton.

14:34As we've been hearing, this is an issue raising important questions about the nation's national security and whether our information is now all too vulnerable to attack. Bridget McKenzie is the shadow minister for Infrastructure, Transport and Regional Development. She shared the federal opposition's concerns when she spoke with Sally Sarah this morning. Is the opposition satisfied with the government's response to this AI incident? I think this incident is concerning on a number of levels. Internally and domestically here, the fact that the reporting mechanisms for unauthorised breaches isn't clear. It's a little opaque that we haven't got something in place. I think the time taken for services Australia to actually report to their own minister, the breach and what they were doing around that. I mean, there's questions as to, I just got on services Australia's website this morning and there are multiple emails on how to contact Medicare as a result of that.

15:40So my question would be how are they sorting through that data? Is it actually an AI system that's sorting through those emails and determining priority, etc. We're in the services Australia system. So concerning that the minister that took so long and I do think that there's undoubtedly been a political decision to delay informing the Australian public. Given last week, ministers across the government knew about the breach and that a decision must have been taken somewhere to wait until the Prime Minister was on the world stage. And for a variety of reasons, the decision to announce the breach there. But I think there's also concerns on the open AI space. It shows I think a disrespect for a democratically elected government that over previous weeks,

16:43they've literally had humans speaking to our ministers, turny generals and others about watering down our copyright laws around our government and our nation's approach to AI and frontier models and the like. And nobody in those human to human meetings chose to disclose this breach. So I think there are significant questions there. Could there be legal consequences for AI companies if their AI agents gain unauthorized access into government or private enterprise portals? Well, my understanding this is something the Prime Minister's task force will be examining. We need to make sure, as I've said, for a long time now, humans need to remain in control of artificial intelligence, not the other way around.

17:44And what that breaches like this show is that things are escalating, intelligence is growing, the capacity to learn and act independently is also growing. And so we need to be making sure the systems we have in place here domestically. But I would suggest across the world need to be fit for purpose. So we don't want to get into a situation where I guess the remedy is such that people don't disclose. Right. We need to have systems in place where we are all working together to make sure artificial intelligence serves humanity, not the other way around. The government is developing national AI standards. In light of this incident, does the opposition agree with the government that regulations need to be finalised by the end of the year?

18:46Well, again, I'll leave our responsible shadow minister to deal with the specifics of those questions. We've been very upfront and open with working with the government, set up a parliamentary joint committee where both the opposition and the government work together to be looking at not just regulation, necessary regulation for artificial intelligence here at home, but also those questions around citing of data centres, how they'd be powered, etc. I mean, I think this is going to be an exciting opportunity if we can get the settings right. And around powering those frontier models and using data centres, community consent is important. But so we've got to ask the question, what are we going to get out of it as a nation? And maybe it's things like lifting the moratorium on nuclear power so you can have affordable, reliable power for these things that zero emissions.

19:46So there are some exciting opportunities and we'll be working with the government to get it right. Bridget McKenzie is the shadow minister for infrastructure, transport and regional development. The development of AI has seen a race between the United States and China for dominance over the emerging technology, something that's seen US President Donald Trump give short shrift to calls for a slowdown and stricter regulation. In fact, artificial intelligence has been one of the items on the agenda today at a historic meeting at the White House with his Chinese counterpart Xi Jinping. The trade dispute between the superpowers is up for discussion too. And for more on that, Sally Sarah spoke with John Zin, a fellow in the China Centre at the Brookings Institution. He's also a former director for China at the US National Security Council under President Joe Biden and he was a China expert at the CIA. What's your read on the overall relationship at the moment between the United States and China? I would say that the overall relationship now is kind of stuck.

20:49We're not seeing an escalation in tensions and I think that this kind of head to state head of state to state visit is designed to do exactly that. And we also don't see a meaningful regression towards a more competitive posture for me that are signed. I think the fact that this truth has been extended just for a few months rather than for a longer period of time, illustration just how stuck the dynamic is and how even with all the fanfare and pageantry that we're seeing here in Washington today, how hard it is to make forward progress even to agree to something that seems to be mutually beneficial for both sides. John, often during this second Trump presidency, we've heard the narrative of China trying to put it forward itself as ordered and predictable as there's been unpredictability politically and in foreign policy in the United States. How do you view that narrative? That has certainly been the posture that Beijing has struck. And it's very much opposed. And it's opposed that they also struck during the first Trump administration. I'm very struck that they have basically cut and pasted many of the talking points that

21:52they used the first time around. I think the challenge for the United States is that because the trade war from the first year of the Trump administration were omnidirectional rather than just focused on China, there is a certain resonance to those points. And I think one of the interesting things about Xi Jinping's third-term and office is that there has in fact been so much policy continuity. So I think it's very unfortunate for the United States. And I think there is a real missed opportunity because there seems to be a growing convergence of interest between the United States and its allies and partners such as Australia, Japan, and I would argue also the EU on China issues and a greater appreciation of the challenge posed by China. But the administration has been more focused so far on picking fights with its allies than with its rivals. John, the public posture is one thing. It's interesting at the same time that this so-called race on AI is unfolding. What does that tell us about what's really going on in the US and China currently? Yeah. It shows that there, you know, beneath the surface and beneath the facade of diplomacy,

22:54there is real competition going on between the two sides. And I think the Chinese for their part are very focused on catching up to the United States. And even here in the United States where there has been so much anxiety and concern that really exploded over the past few weeks or so after the hugging phase incident, you know, the one common denominator from, you know, as diverse a crowd as President Trump and Dario Moda, the CEO of Anthropica is that they all want to see the United States remain ahead of China when it comes to this so-called race over AI. I think the real challenge is defining, you know, what exactly the race is over and what's the NSA. It's going to be a focus in the US on the frontier, but the Chinese are making a lot of strides both in trying to catch up to the frontier and also accelerating adoption across its economy. John Zin is a fellow in the China Center at the Brookings Institution. Thanks so much for listening to all those interviews from Sally Sara produced by the ABC Radio National Breakfast Team. Remember, you can hear the radio show from 6am every day.

23:55The breakfast wrap today was produced with help from Pip Cook and I'm Nick Grimm. We'll be back with more stories from Radio National Breakfast next week.