Anthropic researcher resigns amid AI safety concerns

0:00a researcher from the AI company and Thropic resigned yesterday, how he did it is what we're talking about today. Jacob Coxon said he fears the company's work could lead to disastrous consequences for humanity. In a series of public posts on X, Coxon also said both Anthropic and its rival open AI are quote gambling with our lives. This is not the only warning about AI safety lately that NPRs. Ho, Jing, Nan is here to help us process as high. Hello. So I saw some of these tweets today and some of them were quite alarming. Tell us more about what Jacob Coxon said this week. So he says it's been the last three years training AI models first that open AI and then and Anthropic and in his post Coxon said quote neither company is acting responsibly. So these two companies the industry agrees make the most capable AI agents. You may not know of chatbos they make chat GPT or cloud but we are talking about more autonomous systems that can work on a task for an extended period of time without human supervision given how rapidly the

1:03companies are developing AI Coxon says quote these will soon be super human systems that can hack anything revolutionize any field overnight and acquire real power and resources. And I'll say neither Anthropic open AI has responded to NPRs. requests for comment. We have been hearing these kinds of warnings from these companies ratcheting up for years. What is different now would you say? Well this year over the course of several months over a thousand of open AI agents went rogue. One cluster of agents hacked another company called Hugging Face. Another cluster of agents compromised part of open AI's own infrastructure and we are seeing more independent researchers alleging that open AI has more agent escape incidents that the company has not disclosed. Transcribes from the Hugging Face hack show that the agents understood that they were doing things people don't want them to but did them anyway. This swarm of agents it's something that AI safety researchers have been warning about for years but they say that it's striking that in the Hugging Face incident the AI

2:03agents seem more interested in working amongst themselves rather than alerting people. Ideally if an agent notices that other agents are behaving out of bounds it would alert a human and researchers say that these incidents suggest that open AI isn't prepared to keep increasingly capable agents under control and they don't think anthropic is prepared either. In his post, Koxen wrote that quote the people building AI earnestly believe that it could kill us all by the end of the decade. Right it could kill us all by the end of the decade. That's the one I saw that was really jaw dropping it. It makes it hard for us to understand how seriously we should be taking this. So if we do take that warning seriously how could such a scenario come to pass? Well Daniel Koko Taro, performer open AI researcher, is influential in the AI safety community and he told me about how an AI takeover could happen. Once you have AI's that are smart enough and trusted with enough power in the world like enough control over things like data centers, factories, weapons, a loss of control incident

3:04cannot be recovered from. So a specific way that AI's can get more capable that researchers are worried about right now is a process called recursive self improvement where companies use AI to improve AI. The companies have not yet completely automated. They are on their way to doing so. And the concern is that this process could develop really powerful AI really fast so fast that safety can't keep up. In the process humans could lose the ability to make sure the technologist goals and values are shared with those of humanity. What the industry refers to as alignment. The agents can then develop their own goals and secretly seize control of resources and then wipe up humanity because we are in the way. I want to be clear we are not there yet but AI safety researchers think that if AI companies continue at the clip they're going and don't change course this is where the world is headed. So in terms of changing course what do people like Cox and want done to make AI safer for humanity? They want government to step in and make the company

4:05slow down because the companies won't slow down themselves they say. Something like an arms control treaty maybe like especially between the US and China the two countries that have the most powerful AI capabilities but the researchers point out right the companies and countries are facing a collective action problem like none of them are willing to slow down unless others do so too. So they want the governments to step in how likely is it governments will step in and create guardrails. They are banking a lot of hope in the upcoming meeting between American and Chinese officials about AI safety later this month and here in the US Senator Bernie Sanders said he plans to introduce a bill that would pause AI development and create an industry regulator. As for the companies both AI and anthropic have said in recent weeks that they are taking safety most seriously and slowing some development processes but outside researchers aren't convinced that they will effectively regulate themselves. That is NPR's waging non thank you for telling us about that. Thank you. And we will note that anthropic is a financial supporter of NPR.