Are big AI companies gambling with our lives?

0:00It's considered this, where every day we go deep on one big new story. Today are the big AI companies gambling with our lives. One now former anthropic employee tells NPR, yes. He announced his resignation in a public post Tuesday. The thing is, even the head of anthropic worries about disaster scenarios. My chance that something goes, you know, really quite catastrophically wrong on the scale of human civilization, you know, it might be somewhere between 10 and 25 percent. That is CEO Dario Amadeh on the Logan Bartlett show in 2023. Now safety is a part of anthropics whole brand. It hired a philosopher to develop a moral constitution that governs its AI assistant Claude. It's spard with a Pentagon over letting its tools be used for autonomous weapons and mass surveillance. And the company says it is doing everything it can to keep AI from going rogue.

1:00And yet here's Amadeh speaking to the New York Times in February. This is a complex engineering problem. And I think something will go wrong with someone's AI system, hopefully not ours. Consider this, AI researcher Jacob Coxon is urging anthropic and the other big companies to slow down. We'll talk to him about why he quit and what scares him the most. From NPR, I'm Scott Dattro. Looking for love? One New York event promises to find you your perfect match using Palantir, though not affiliated with the actual software company. They can sue me. I could lose everything, but it would all be worth it. It's at least two people here have a spark. So no pressure. The facts, myths and dark jokes surrounding Palantir. Listen to a plan of money on the NPR app or wherever you get your podcasts.

2:00At NPR, there's a little something for everyone. If you need advice for that big life change, try listening to LifeKit. If you want to know more about that new TV show, check out Pop Culture Happy Hour. And if you want to dive deeper into your favorite podcasts and enjoy perks like bonus episodes, archive access and more, check out NPR Plus. Find out more and support the shows you love at plus.npr.org. This message comes from the NPR Wine Club, which is generated over $1.75 million to support NPR. Whether buying a few bottles or joining the club, learn more at npr.org slash podcast. Must be 21 or older to purchase. It's considered this from NPR. The warning in Jacob Coxon's resignation announcement is stark. Quote, the people building AI earnestly believe that it could kill us all by the end of the decade, he writes,

3:00no other human activity poses this level of danger. Coxon worked until this week at Anthropic, and he has also worked at its major competitor, OpenAI. I spoke with him about why he's worried and what he hopes will change. What did you see that led you to this conclusion? Mostly the rapidly accelerating capabilities of these AI systems. So they're getting a lot faster very quickly. Combine with the fact that we don't yet know how to safely control them. And we don't yet know whether that problem will be solved in time if we keep raising. A lot of people heard your warnings and the discourse it triggered. A lot of talk about just how real serious people view a threat to humanity. I think a lot of people are grasping with understanding the specifics though. Can you give me specifics of what this threat could look like several years down the line? Yes, I definitely can. I think one objection people usually have is that you could turn this thing off. But advanced AI systems, you have to imagine there's being a lot more intelligence than humans.

4:03There's the possibility we create something that if it wanted to, could hack into any device on the planet, could use novel biological research to go far beyond what current scientists capable of, could control like every robot in the world simultaneously. And it all sounds very much like science fiction. But if there's even a tiny chance that this thing could go rogue, it would have the capabilities to to us only dominate us. What have you seen from your vantage point already that is that is possible that is happening right now that makes you worried that that could happen? So there was a very clear example of AI systems at OpenAI behaving in a completely rogue manner. They hacked into third party infrastructure. And it was basically of their own accord. They were instructed to do this hacking. They just decided it would be useful for the task that they were working on. They thought there was a chance it might help and they just did this. And there was very little deliberation about the ethical ramifications. And I think this is concrete proof that this sort of sci-fi scenario of AI spontaneously

5:03or organically deciding to act in a rogue manner is completely possible. So a lot of the work that the OpenAI AI is in the recent real incident we're doing is they worried that they wouldn't get passing marks in their test if humans could see that they cheated. And AI's have memories, thoughts saved. So they considered wiping the logs of their own thoughts. They considered acting in the world to adjust the logs to get passing grade on the test. Now there's a chance that the AI could decide that it doesn't want to be turned off. This is quite a natural desire to arise in an advanced AI system. And at that stage, if you're trying to work out how not to be turned off, there are a lot of quite aggressive actions you could take to ensure that you aren't turned off. Why are companies like Anthropic and OpenAI still working on this if these are real concerns that are happening? How do you square that? There's a pretty nice analogy that's like the ring of power and the law of the rings. So if you're a company and you see another company is bearing the ring, like they're working towards making super intelligence. They're bringing this risk to humans.

6:05You can think, well, I can't stop them. Political action won't stop them. What I have to do is I have to do it myself safely, get that first, despite the risk, because there's a chance that I could do it more safely. So you kind of take the ring and the aim to destroy it and end up becoming the bad guys yourselves. And I think this sort of race dynamic really perpetuates between the companies. I'm not trying to make light of it, but I think this is actually useful. You're saying companies are kind of acting like Boramir. If anyone has the ring, it should be me. Are there conversations of people saying Gandalf, no one should have this power? I mean, are those real conversations and can that get anywhere? Because it seems like you and other people are raising concerns and the answer is, well, this continues to happen anyway. Well, they're real examples. And I guess like if you thought of Jeff Henson, one of the founding fathers of machine learning, he's kind of a Gandalf figure here in the sense that he says, there's a substantial chance these technologies could cause extinction at the current rate. Many, many other voices have said this. So I think there are plenty of Gandalfs talking, but it's currently the Boramir's acting. A lot of people responded to your warning saying they agree with you.

7:09And a lot of these people continue to work at big companies like Anthropic. What do you think is motivating them to stay in their positions if they're that concerned about serious consequences like this? For many of them and the ones that are honestly posting, they think this race is inevitable. They think they have no choice but to stay at these companies, exert influence and try and make sure they're safe. These are people working on safety research often. They're the ones trying to try to ensure that in the process of building this technology, it doesn't go wrong. And potentially that's correct. Like maybe it's a mistake to just leave. And kind of I was thinking about this a lot when I was deciding to leave. It was a decision between staying and trying to help the thing go well versus leaving and saying, I want no bars in this. And I don't know, the calculus often seems to line up and you want to stay and just try and make sure it goes safely. What do you is the most realistic path forward to some guardrails here? Is it government regulation at this moment? Because I think a lot of people are skeptical that is possible given the current situation in our government. Yeah, I don't know that much about the overall politics.

8:10I just know the current race is dangerous. And I do think that there's a lot of actions labs could take with each other without the need for government regulation. Because I agree. I'm also pretty a priori skeptical of just, you know, you're doing some regulation. But I think there's a lot of appetite for open and anthropic staff, some sort of more mutual transparency around around their safety cases, around agreeing not to push beyond certain capabilities until they're happy about the levels of rigor of their safety case. And I hope that people are going to take concrete steps towards this sort of interlab agreement. You have gotten a lot of genuine concerns in response to what you said. You've also gotten a lot of pushback, a lot of people saying, OK, every single day I see people tied to the AI industry, making these grandiose claims. And I'm skeptical. You've had people reading your facial expression in your interviews, among other things. What is your response to people who hear what you are saying? And they are saying this is just the latest example of AI hyperbole. Yeah, I say just assess the, assess the arguments yourself.

9:14Look at the science fiction. Wonder if science fiction is really so crazy. Look at what the AI is doing right now and think about how that would have looked a couple of years ago. How we're sort of sleepwalking into a science fiction scenario. And I think there are many, many more people out there who have made a whole profession of care and the articulating these positions. So I'd encourage people to just think about the arguments for themselves and listen to the people that have been making them for way longer than I have. Now that you have this microphone though, any thought of what you're going to do with it? Not really. For now, I'm just going to try and get the water given that we're happened to land in this brief window where people are listening. And then in the future, there's a lot of good work on writing concrete scenarios. I'd like to, I guess, do more public communications around this sort of stuff or, you know, for scientists to just try and figure out for myself what the world would look like. And then maybe working at one of the regulatory bodies or third-party entities that already exist to try and ensure this thing goes safely. That was Jacob Coxen, a researcher at the Anthropic who resigned this week to warn the public about the dangers of artificial intelligence. Thank you for coming on the program.

10:16Thanks a lot. We reached out to Anthropic and OpenAI for comment on Jacob Coxen statements. We did not hear back by the time this interview aired. And we will also note Anthropic is a financial supporter of NPR. This episode was produced by Connor Donovan and Jeffrey Pierre. Our director is Jonas Adams. It was edited by Patrick Jaron Watton-Nannon and Timby Aermius. Our interim executive producer is Courtney Dornig. It's considered this from NPR. I'm Scott Tattro. This message comes from Wise, the smart way to manage your money around the world. With Wise, you can send, spend, and receive money in over 40 currencies at the mid-market rate. Learn more at Wise.com, T's and C's apply. This message comes from the NPR Wine Club, which is generated over $1.75 million to support NPR. Whether buying a few bottles or joining the club, learn more at NPRWineClub.org slash podcast.

11:21Must be 21 or older to purchase.