Iran and AI on the battlefield

Deep fake porn didn't come out of nowhere.

It was allowed to spread while governments dragged their feet and tech companies shrugged.

I'm staring at myself in this video that I know I haven't made.

This is what it looks like to feel violated.

This season on Understood.

If you follow the trail, who does it lead to?

These images they would like hunting me and the biggest platform was Mr. Deep Fakes.

Understood. Deep fake porn empire.

Available now on CBC Listen, or wherever you get your podcasts.

This is a CBC Podcast.

Hi everyone, I'm Jamie Poisson.

For decades we have been hearing about the possibility of AI-driven warfare.

Now it's here.

Inthropics, AI platform Claude has reportedly been central to America and Israel's war on Iran.

Within the first 12 hours, there were more than 900 strikes, including one that killed the supreme leader,

Ayatollah Mane.

An air strike on a girl's elementary school killed at least 165 people, mostly students.

Today we're talking about AI military capabilities.

How companies like Anthropic and Open AI have become or are on their way to becoming

deeply enmeshed in the military.

And what happens when these companies and governments start building systems that help decide

who lives and who dies in a war?

I'm joined today by Heidi Klack.

She is the chief AI scientist at the AI Now Institute and an expert on AI safety within

defense and national security, including in autonomous weapons systems.

And she previously worked at Open AI.

Heidi, hi, thanks so much for making the time.

Thank you for having me.

And it's really great to have you.

So let's start with what we know about how AI is being used in the war with Iran right now.

What role is it playing?

So Claude is currently being used as what we call a decision support systems,

which means that it brings together a lot of different types of data that has been gathered.

So you're looking at satellite images, social media feeds, intercepted film communications,

and then it uses them to make recommendations, including target recommendations,

prioritizing them, and even providing coordinates for those targets.

They're also likely used in other phases of the kill chain, so that it includes things like

intelligence, gatherings, surveillance, and even collateral damage assessment,

as sort of have been confirmed by research reports.

And this is all happening, as I understand it, because of this pairing

with the military's maven smart system, which is built by the company Palantir,

which is mining all this data, including classified data.

And then it's been meshed with Claude.

And can you just describe to me what's going on there?

Yeah, so Palantir is a data analysis company,

and they work a lot on intelligence and surveillance.

And back in 2024,

Anthropic and Palantir had an agreement where essentially Anthropic's model

that being Claude within power sort of this recommendation engine,

which takes all of the enormous amounts of data that Palantir has sort of analyzed,

collected, and sorted through, and then feed it to an AI to then make some sort of decision.

And so I mentioned that strike on the girl school in Iran.

The Pentagon is denying targeting civilians.

Israel says that there was no IDF operations in the area.

But is it possible that AI was used in some form here?

I think that's actually a very good question,

because it brings the point forward that AI is actually being used to evade accountability.

And that's because when you're using these types of systems,

it makes it difficult to distinguish if some attacks were in fact deliberate,

due to indulgence failures,

or due to the lack of AI accuracy,

or the parameters that were even set for these systems,

in terms of how many casualties they were willing to accept.

So it actually muddies that accountability altogether,

while also obscuring what the LLM is actually doing here.

And so it's very difficult for us to say if this was due to an AI mistake,

or if this was deliberate or not,

and that's exactly why a lot of militaries are using AI, it obscures that.

Right, because I had been reading that the school was at one time a base for the IRGC,

right? And so is it possible that that information was just scraped

from some kind of system and then analyzed using these tools?

It is a possibility, but it's impossible for us to know.

And also when we're talking about these systems as well,

a lot of the times they provide these recommendations,

and it could be on outdated intelligence.

So it could have been a combination of both intelligence failures,

and also the model really not understanding that that data was no longer relevant.

And that's exactly why these models shouldn't be used for targeting,

because they have a very difficult time making these types of distinctions,

where a human might be able to.

It also very well could be that this was deliberate or not the fault of the AI.

Right, but as I said, it's very hard to know.

The

Claude through this partnership with Palantir was also used by the Pentagon

during the U.S.

Raid to capture Venezuelan President Nicolas Maduro,

according to the Wall Street Journal,

though there is not a lot of details on the precise role there either.

And just like what other examples of military AI use do we know about in recent years in the U.S.

or elsewhere really?

So there's a lot of different types of AI,

and I just like to always make it clear that we've had AI be used in the military,

including within the U.S.

maybe even since the 1960s, as far back as that.

That's when a lot of research started on using these types of systems.

But those are very different types of AI than what's being deployed today,

in that being large language models, right?

Even Maven initially used very different types of AI

before large language models came in,

and then they sort of had the partnership with Anthropic.

And the thing about these different types of AI is that

military purpose-built AI is what we call sort of the former category.

It tends to be very task-specific.

It's trained on specific sets of data,

and very specific sets of tasks.

Allolums, on the other hand, or gerryntive AI,

are very general purpose,

and then they're kind of like fine-tuned to be repurposed for military purposes.

But the thing to remember is that military purpose-built AI,

or task-specific AI, tends to be more accurate

than general purpose models, like elements.

Allolums have an incredibly low accuracy rate.

You're looking at a lot of hallucinations.

You're looking at something like 25% to 50% accuracy,

and yet they're still being deployed.

And that's not to say that other types of AI weren't that accurate either,

but at least they were, you know,

instead of us going towards a direction of maybe stepping away

or understanding a little bit more how to responsibly deploy AI in the military.

Instead, we went to the complete opposite direction.

We're deploying something that's even worse than what we had before.

So I would say if we're talking about different types of conflicts,

yes, we have seen AI.

There's been a lot of investigations on drone strikes in Afghanistan,

and Syria, and Iraq even.

But that's very different from what we're seeing today,

where we're essentially having a system

really make a final decision on everything,

and they're just simply not capable of doing that.

Right, and just to be super clear here about what you're saying is

it's the same kind of system as these chatbots that we're using.

Yes, right?

Like OpenAI's Chatchee BT that regularly hallucinates and gets stuff wrong.

Exactly.

Yeah, what they actually do is they take these everyday models

that you and I have been using.

And they input new data on top of it.

And essentially, you basically have the same baseline model

with a bit more context on military targeting, for example.

But it's still the same type of hallucinations.

It's still the same lack of accuracy.

So these aren't improved model as they would like us to believe.

We've talked on the show before about this really explosive reporting

from the Guardian and 972 magazine that revealed through

intelligence sources how Israel used AI at one point

to identify like 37,000 potential human targets

based on their apparent links to Hamas during the war in Gaza.

This AI system was called Lavender, the reporter's

interviewed people who said they would take the information

and drop a dumb bomb on a target often killing an entire household of people.

And is this what we're talking about here?

Was this generative AI?

So actually Gospel and Lavender did not rely on large language models,

but they used different other types of AI to essentially

have the same outcome.

But after that, we did receive confirmation

that both GPT-4 and Google Gemini, which are language models.

They were then eventually used to generate invalidate targets.

And we started seeing hints of that when

we saw the cloud contracts with Google and Microsoft

that were ramped up after October 7th.

So even though they had Lavender and Gospel prior to that,

as soon as they saw the opportunity to use large language models,

they went ahead sought out these tools to deploy them very quickly.

And regardless of the type, the AI algorithms are all looking to do the same thing, right?

They're looking to generate as many targets as possible.

And they all ingest similar types of information

and then they give recommendations.

So they're used towards the same purposes.

And as I mentioned, both types of AI really have accuracy problems here.

And just big picture, how would you say the use of these LLMs

is changing modern warfare, what we're seeing today?

I'd actually argue that it isn't changing it.

I think when you keep in mind the abysmal accuracy rates

and the speed of LLMs, it's almost just a high-tech version of carpet bombing.

And what we're actually seeing is that AI is being used to evade accountability.

The very use of these systems, like with the school,

will make it very difficult to say, okay, who's fault was this?

And so when you're looking at these types of issues together,

the evasion of accountability and the fact that these systems aren't accurate,

we're just going towards a world where we're no longer following the traditional legal rules of war.

And I think that's very, very concerning.

So to me, we're just seeing us move away from that accountability,

from understanding the type of targets that we have,

from us understanding the decisions that are even being made.

Because with AI, we don't know why some of the decisions are being made.

So it's not to me, it's not an advancement, but rather a regression.

Right, right.

Like I've just seen legal experts talk about how they worry that what this will ultimately result in

is rubber stamping automatic strikes with no meaningful human oversight.

Would you say that that's a pretty good summary?

Yes, and I think it's really important to remember here, especially if you're looking at specifically

anthropic, because they had this fallout with the Department of War.

A lot of people are talking about, well, anthropic has a strong red line,

because they're talking about, they don't want their models to be deployed for autonomous weapon systems,

and they don't want less human oversight.

I don't consider that a moral high ground, because actually,

in practice, the difference between decision support systems, which is how they're currently being used.

And AI-driven autonomous weapon system is this human interloop that you're talking about,

and that's impacted by what in our field we call automation bias.

And automation bias is this idea based on decades of research showing that humans often

trust the recommendations of algorithms without corroborating with other sources or checking

if that recommendation is correct or not.

So it does, in the end of the day, you know, end up being rubber stamping.

And then again, when you combine that with the knowledge that these things are very accurate,

you're looking at the normalization of this technology that shouldn't be used for targeting.

And I think that has huge implications of the worst to come.

And we already have a very good example of that already,

which is what happened in the Tesla, where these types of AI systems were also used by the idea.

Right, right. In Gaza, I know a lot of people call the rest as well, just for people listening.

Yeah. What's your worst fear here?

Like, do you have a nightmare scenario that keeps you up at night?

Yeah, I think we are in the nightmare scenario when it comes to military use.

I think something I'm really worried about is also giving these models access to nuclear weapons.

And when we're talking about things like autonomous weapon systems and then we're talking about,

you know, we saw scale AI, which is a different AI company recently land a contract on nuclear

command and control. It really makes you question what's the next step here? We are already in such

like a terrible, you know, worst case scenario in terms of giving these models ability to target,

right, with very little human oversight. And now we're seeing just the expansion of that.

The fact that we're even having the conversation about autonomous weapon systems is what's

scary for me that no one took a step back and say, wait, hold on, they shouldn't even be used for

decision support systems. And here we are talking about autonomous weapon systems. And now there's

nuclear command and control. And I think, you know, we're actually deploying these systems in a very,

very, very reckless way. And it seems like there's no stopping. And, you know, a lot of people are

hoping that Anthropic is a hero here and they'll draw the red line. But at the end of the day,

they're back on the negotiating table with the Pentagon as of today. And at the end of the day,

Daru Amadeh, who's the CEO of Anthropic, came out and said, oh, no, I'm not opposed autonomous

weapon systems. We just don't think they're reliable enough. And we want to talk on developing them,

you know, so they don't have those red lines. So if the governments aren't doing, and the companies

aren't doing it, this puts us in a very, very bad position of what's the context.

You know that feeling when you reach the end of a really good true crime series,

you want to know more more about the people involved where the case is now. And what it's

like behind the scenes, I get that. I'm Kathleen Goldhar and on my podcast crime story,

I speak with the leading storytellers of true crime to dig deeper into the cases we all just can't

stop thinking about. Find crime story wherever you get your podcasts. I want to dig into this whole

Anthropic Open AI kind of fight, I guess, with you just in one minute. But first, I wonder if I

could ask you like what I think might be an incredibly obvious question, but also I think

might be helpful. Like what would an autonomous weapon system do?

This is actually a really good question because I think a lot of people think of killer drones.

That's the first thing that they think of. And they, I mean, I am thinking of killer drones.

Exactly. Yeah. There's many different types of autonomous weapon systems. In fact,

you have organizations, international organizations that consider minds autonomous weapons systems,

right? They could look like many things. When you're talking about drones, that's a very different

type of AI. You need an AI with specific skills like object recognition. These are not general

purpose models. And then based on that and, you know, some data that's fed into them, they then

sort of lock in and target something. That's very different from how we're using frontier AI

models, which one on these enormous cloud systems in just a huge amount of information.

And based on that, give recommendations. And if we're thinking about a targeting recommendation,

you can then use any type of weapon to strike that target. Right. So it could be a drone. You could

send a drone off after that target has been identified. It could be a missile. If it's an

infrastructure target. So there's many different levels of autonomous weapon systems. And I think

most people really have focused on this, like, you know, killer drone, drone forms. But that's

actually not where frontier AI or cloud is being used. They're using on a much bigger scale

where they're given many different types of targets. But typically a human then takes that

recommendation and decide, right? Like, I'm going to send a missile after that, right? Or I'm

going to send a drone there. But the idea is then to eliminate that human that when we're talking

about frontier AI, to eliminate all that together. So whatever recommendation is given, the model

itself can even decide the type of weapon that's being used. We saw scale AI sort of give demos

on that using large language models. Like, it's a different company that also had a lot of contracts

with the Department of War. But, you know, it's like, what would be the best type of weapon to use

for this recommendation? And so the idea is to just eliminate the human from that loop and just

to automate that process. Got it. And just make it so that maybe a human just, as we were talking

before, kind of rubber stamps it. Yeah. Okay. So given that, I mean, given this request that the

Pentagon had of Anthropic, that they wanted the company to remove restrictions on things like

autonomous weapons and also large scale surveillance. And so we have said to the Department of War

that we are okay with all use cases, basically 98 or 99 percent of the use cases they want to do

except for two that we're concerned about. One is domestic mass surveillance. They're worried

that, you know, things may become possible with AI that weren't possible before. Case number two

is fully autonomous weapons. This is the idea of making weapons that fire without any human

involvement. I just wonder if you could tell me a little bit more about what that told you about

what the Pentagon ultimately wants to do here, what direction they want to go in with AI.

I think they very clearly want to use these technologies as an olive eye for whatever actions

they want to carry out. Like I said, when we go back to the point that AI really helps you

evade accountability, it makes people question, was this deliberate that they do this on purpose?

And it also makes it very easy to say things like, well, the AI determined it so must be true,

right? There's I think a group of people who view AI decisions as being more objective than

humans and that's not the case in any way. And so it's much easier to hide behind AI decision

and not have to justify them. So I think when they want no guardrails whatsoever, it generally

shows that they want to be able to use these systems in whichever capacity they feel like.

On the surveillance stuff, the Atlantic reported that Anthropic was told the Department of

Defense, I guess the Department of War now wanted to use Claw to analyze bulk data collected

from America. So everything, I think you type into a chatbot, right? All your search or credit card

history. And just what could be done by the military with that level of information? So I think

it's really important to note that these models are what we call dual use. So they can be used for

civilian purposes and they could be used for military purposes. And when you essentially

start thinking about how they already started using these model for intelligence gathering,

right? And intelligence sort of analysis, you start to think, right, this can also be used on,

you know, the citizens of that country themselves, because it's the same feature. So if you collect

data from different sources, so that thinks like data brokers, location data, internet habits,

and that's not just the things you're using the model for. That's really commercial data you can

buy from anyone from data brokers, as we call them. And then you train or input these specific

data points into the Netherlands, you can then draw inferences on whom these individuals are

and track them accordingly. And again, as I mentioned, it might not be accurate, but you might be

making decisions off of that, which again, is really, really problematic.

The US Undersecretary for Defence, who is negotiating the deal with Anthropic,

was defending the position. And he said, well, at one point, he said, quote,

At some level, you have to trust your military to do the right thing. And if anything,

we're the biggest, you know, organization in the world with the most rules of any organization

in the world. But we do have to be prepared for the future.

He also made the argument that what Anthropic was afraid of or what people were afraid of

was already barred by the law. And just, um, does he have a point there, I guess?

I would say that the Department of War conducts its own legal reviews and compliance with

international law. And they likely see that Anthropic is overstepping its role in these

determinations, including their legal judgment of whether or not to blow these systems.

However, from an international law perspective, there's going to be different interpretations of

what the lawfulness and predictability of these AI based autonomous weapons systems

than difference from how the current Department of War currently chooses to interpret them.

So the international community output is necessary here. And we haven't seen any consultations

either from orgs like the UN or the ICRC on these types of decision, especially when you have

Dario raising concern that AI is too unreliable. And that would support an interpretation actually

that LM based autonomous weapon systems are not in line with international law. And either way,

we shouldn't be at the whims of a private corporation red lines on whether or not this dangerous

technology should or should it be deployed. But I think it's at the end of they also

important to remember that there is international law, right? And it's not just is a private corporation

versus the Department of War, right? There's already restrictions on what types of autonomous

weapons systems can be used. And currently when we're looking and when Dario admits himself that

these bottles are reliable, it says a lot about their legal status if that makes sense.

Yeah, though I guess some people might be listening to this right now and feeling like maybe

they don't have a lot of confidence in international law at the moment, just state that obvious fact.

Look like I take your point that you know you don't want to make unthropic out here to be the hero

and the CEO to be the hero because they are back at the table now negotiating with the Pentagon.

But just I wanted to ask you about the threat to declare them a supply chain risk.

Yes. And then also obviously that coming to fruition, right? This label has never been used

against the US company as I understand it and would bar the company from any work with the government.

And just like how big of a deal is that for a company like anthropic? I mean it's quite a

lever for the government to pull. Definitely, definitely because ultimately this is the first

frontier AI company that's been working with the military and they're embedded with them.

So to suddenly use that label clearly doesn't align with what's already been happening.

But on the point of supply chain, I actually do believe that all LLMs are a supply chain that

to national security and defense and a lot of people internally in the military also believe

the same things. And you know, this is about the companies themselves being foreign adversaries

in any way, which their American companies are clearly not. But this is about the nature of

LLMs themselves that I think is really important for people to understand is that they're trained

on the open Internet and publicly available information. Again, very different from purpose

military build models models where every data point and software decision is traceable.

So as a result of these LLMs being built on an unprotected supply chain, there's a lot of new

and undetectable attack vectors, which include things like poisoning web training data sets or

building backdoors into these models, which actually may intentionally or inadvertently lead to

the subversion of their behavior, which includes in military applications. And this isn't something

that can be patched. And in fact, a study by anthropic themselves showed that you only need about

250 malicious documents to produce a back door into one of these models. And so when you consider

that with the news that China and Russia have been compromising online information to influence

the answers of large language models, it's likely that these backdoors already exist.

So I think it's very difficult for me to speak on the department's award decision of why they would

choose to say this about anthropic, but I actually do think large language models as a whole do

have supply chain issues. I'm, I'm reticent to like repeat what people have just heard themselves,

but I just, I do think it's really important to just put a line underneath this that essentially

what you're saying here is that it's not anthropic that is the threat to national security.

It's the data that all this stuff is trained on and that an enemy of the United States,

you mentioned China or Russia, can mess around with the data and it can affect the outcome. Exactly.

What did you make of open AI's decision to kind of swoop in and make a deal with the department

of war after this falling out with anthropic? Is that deal materially different from what anthropic

had wanted to do? You worked at open AI, it would be very curious to hear your reaction to this.

Yeah, I think it was very clear to me that it was not the same deal, although they claimed that

it was. It really wasn't, especially when you're familiar with military contract and these like

open-ended words are being used, right? And so it's very different language from what anthropic,

what's fighting for and initially they came out as saying, oh, we just managed to get the same deal

and then it ultimately came out that that wasn't the case, right? And so it seems that they threw

anthropic under the bus while also trying to sort of benefit from the same PR and also in their

announcement because they talk about a safety stack and this is what I work and I work on safety.

I would say the safety guardrails that open AI is referring to are not operationally feasible

because when you have a model and it gives you an output, you can't monitor after that what

that individual does with the output. They can't double check that that's being overseen by human.

It could just very well be taken to then select and engage a target without any further oversight

making it an autonomous weapon, even if this is deployed on the cloud because this is an operational

matter, not a technical one, whether or not you put a human in the loop. Right, I think Sam

Altman is like basically admitted that, right? Like that they don't, they won't be able to actually

know what the department of work was doing with their technology. Exactly. So when they say we have

the safety stack is going to be on the cloud and that's going to somehow limit it, I think that's

very, very much misleading. How do you describe how a company like open AI approaches safety?

I would call it safety theater safety co-option because ultimately a lot of these AI companies

like to use the same terms that safety engineers use and reference my background is safety

engineering, which is we work on safety critical systems like nuclear plants, aviation,

you know, autonomous vehicles where if they fail human lives or at risk, they take a lot of

that same terminology and essentially kind of safety wash everything with it. Every sort of mission

that they have, they eventually like rolled back on like military use, right? That was eliminated

in 2024 when it was initially part of their charter or their terms of service. You know, we have a

lot of safety incidents that occur all the time with people using their models. There's no guard

dwells really. And so I would say for a lot of people, you know, it's they believe they often

believe the message in like, oh, we care and so on and so forth, but they can write as much as they

want. If you just look at the actions of these companies, these AI models are causing a huge amount

of harm. Contracts with militaries are very enticing for AI companies, right? For a couple of

different reasons and could you just lay those out for me? Well, these AI models are incredibly

expensive to train and it's actually likely the case that they're not even breaking even on

being able to generate the revenue needed to keep producing these big models, training them and

deploying them. So when you have the military, that's a really big money pot for you to be able

to make that money back and also embeds you within military and safety, critical infrastructure,

which means that you're then too big to fail. So for them, you know, when there's a huge amount

of backlash right now against the uses of AI, this is a very secure position for them to have

monetarily and also in practice because, you know, as we've seen with the reporting of

Claude, the entire military infrastructure now relies on Claude for this type of analysis

and it's very hard for them to disconnect from it. So although they wanted a ban,

they were like, oh, we're going to try to phase it out and they still use the interrogue, right?

It just shows the type of dependence and this dependence is very, very strategic for them.

I want to kind of go back to state power. You know, we've been talking about how much power

these private companies have in influencing how wars are fought, but when you look at the pace of

AI development, when you see the military interest in this, the money being spent on it,

is one of the real main worries here that what we are witnessing right now is AI becoming a central

tool of state power. Absolutely. It is the perfect tool to concentrate power because you're

collecting essentially all data possible on humans, on our behavior, on anything really, it even

goes beyond humans, right? You're also correcting data on states, on everything and then allows them

to use those same very systems are able to, like that, huge amounts of information be trained on it.

So then make any sort of decision that becomes very centralized to them and there's no countability

because these models are black boxes. No one's now can question why did you make that decision?

Like we're seeing these models be used for immigration rates by ICE, we're seeing

used in the judicial systems and military and yet they're black boxes and now it feels like no one

can question that power. This is an unfortunate side effect of something like these very large general

purpose models. Okay, seems like a good place for us to end. Heidi, thank you. Thank you so much.

All right, that is all for today. Frontburner was produced this week by Joy the Shen Gupta, Shannon

Higgins, Matthew Amha, Lauren Donnelly, and Mackenzie Cameron. Our intern is Riley Cunningham. Our

YouTube producer is John Lee. Our music is by Joseph Shabison. Our senior producers are

Imogen Birchard and Elaine Chao. Our executive producer is Nick McKay-Blow-Cost and I'm Jamie

Poisson. Thanks so much for listening.

For more CBC podcasts go to cbc.ca slash podcasts.