AI: Should We Slow AI Innovation Down?

0:00This message skims from Total Line and More. Summer is still going strong. Stop by Total Line and More for crisp white lines and cold beer. Always at the lowest prices. Spirits are not sold in Virginia and North Carolina. Drink responsibly, be 21. Last week, a researcher at the Artificial Intelligence Company and Thropic announced his resignation on X. Jacob Coxon wrote that his former company, along with OpenAI, where he also recently worked, were acting irresponsibly and, quote, racing straight to self-improving super-intelligence and gambling with our lives. That post went viral. It's now been viewed nearly 200 million times. Then on Saturday and Thropic's own CEO, Dario Amade, published an essay calling on the entire industry to deliberately slow down and also shared some ideas on how to do it. The head of OpenAI, Sam Altman, replied,

1:00I agree with Dario that we need to pace the frontier. Elon Musk and Demis Hassebiz at Google DeepMind agreed too. But President Trump isn't on board. Here's part of what he wrote on Truth Social on Monday. Quote, the only control or guardrails that AI needs is a strong and smart, high IQ president, and the USA has that in spades. He continued, quote, there is a sick conspiracy going on against AI and data centers, and the only one that is happy about it is China, whoever wins AI wins. Well, back in June, we asked what would happen if artificial intelligence became smarter than every human at every task? That's the so-called super-intelligence referred to in Coxon's social media post. But in just a few short months, that conversation is shifted out of the hypothetical with real events that make the existential threat look more possible. So we returned to that debate today to ask what's changed. I'm Jen White. You're listening to the White Aid podcast. Should we slow down AI innovation and is that even possible?

2:02What's at risk if we don't? The stakes are high, we get into it after the break. This message comes from Betterment. Dan Egan, VP of Behavioral Finance and Investing, explains why betterment was designed to take the time-consuming work out of smart investing. I was doing all of these things that were pretty straightforward to implement. I just needed to spend my time doing them. Betterment automates the same practices, so that I know I'm doing portfolio management and goal-based planning without me having to spend hours of my life doing it. Learn more at betterment.com, investing involves risk, performance not guaranteed. This message comes from Rinse. Your dog believes you are magnificent, capable of anything. Your dog has washed you spend hours a week moving fabric between machines and has never once lost faith. The question was never whether you could become the person your dog thinks you are. It's what's in the way. Turns out, just a laundry.

3:03Not anymore. Rinse picks up your laundry, cleans it expertly, and delivers it back while you get on with being magnificent. Sign up today at rinse.com. Rinse, it's time to be great. Let's get into the conversation with Nate Soares. He's president of the Machine Intelligence Research Institute and co-author of, if anyone builds it, everyone dies. Why superhuman AI would kill us all? Nate, welcome back. Thank you. Also, returning with us from the Bay Area is Sias Kapoor. He's an incoming professor at UC Berkeley and co-author of AI Snake Oil. What artificial intelligence can do, what it can't, and how to tell the difference. Siash, it's great to have you back. It's great to be here. And with us from Oakland, California is Natasha Tiku. She's the tech culture reporter at the Washington Post. Natasha, welcome. Thanks for having me. Now, we invited Anthropic, Google DeepMind, and OpenAI to participate in this conversation, but we did not hear back.

4:03Now, I described a little bit of what's happened in just the last few weeks, and the markets have noticed as well by Monday Tech's stocks were falling worldwide. But the debate over the existential risk of AI, it's not a new conversation. So I'm curious to hear from all of you, whether you think this moment feels different if it feels like a turning point, or is this just another round in the argument that the industry has been having for years, Nate? I think it's absolutely a turning point. I think a lot of what we have seen in terms of public awareness comes downstream of some frankly pretty disturbing instance this summer with AI swarms that spontaneously broke out of the labs. So I think this is a very important thing to do against their instructions. I think this got a lot of researchers spooked, and I think this is what led to a lot of these researchers, including Jacob Coxon, who resigned, but including many others who stayed in the AI companies. A lot of these researchers came out and said, we really think that this has a double-digit chance of wiping out civilization, and finally, the world is hearing that,

5:04and taking it seriously, which I think is a big change. So I asked what about for you? I think the biggest change for me was just seeing how recklessly these companies operate. On the one hand, these companies now have over thousands of employees. They are valued in the trillions of dollars. On the other hand, they fail to implement basic control mechanisms that frankly are 10% research lab implements. And so just seeing how recklessly these companies have been operating was a big change for me. Natasha, how are you seeing the conversation shift within the industry itself? Well, we've had a few cycles where potential extinction risk from AI has become national news, but I think on the heels of a populist backlash, you know, against data centers, concerns about the impact on jobs and on education, on child safety. I think it's resonated in a way that it hasn't in, you know, during those previous cycles. Now in his public letter, Anthropics CEO Daria Amadeh, wrote that, quote, since roughly this summer,

6:04AI has been advancing drastically faster. Nate, what's driving that development? You know, I think it's reasonable to model AI as increasing on an exponential, which is to say that it doubles. I think the doubling period is about every four months, depending what you're measuring, which means, you know, it goes one than two, then four, then eight, then sixteen, then thirty-two. And sometimes even following that smooth curve gets you these really big jobs. So it's not so much that we have seen some sudden new insider improvement in AI. It's just that we are starting to get into the very rapid phase of growth here. And, you know, who knows if this exponential will continue forever? My guess is it will not, but also who knows how long it will last. Well, there's also concerns about risks of what's called recursive self-improvement. That's when AI is able to build next generations on its own. Now in his public letter, Anthropics CEO said that this is happening, and it's rapidly accelerated AI advancements since roughly this summer.

7:04Siong, I mean, what do you think about the risks of recursive self-improvement? I think on one hand, I agree with Nate's, sort of, diagnosis that, you know, AI improvements have happened rapidly over the last few years. On the other hand, I should also note that they haven't really happened very smoothly. We've seen this notion called jaggedness, where AI continues to get better at some subset of tasks, while being pretty bad, honestly, at another. For instance, anyone who's used chat GPT or Claude knows that they have all of these weird quirks when they're writing things. They're really poor writers. And I think the difference comes down to whether the task that you're trying to get AI's better at is verifiable, whether you can at the end easily say that the task was solved correctly or not, or whether it's, and whether it is, sort of, more creative or requires more judgment or is more open-ended like writing. So what we've seen is, even when it comes to AI's doing AI research,

8:04the set of tasks that it's gotten very good at are precisely those which require less creativity and judgment. They are sort of straightforward to assess, and that's why we've seen these dramatic improvements in software engineering and mathematics, which are verifiable, while not really seeing those improvements in other areas, including areas of recursive self-improvement, of which still require creativity and taste and judgment from human experts. Well, the anthropic researcher who's post-win viral told WIRED that his former colleagues working in anthropic use words like endgame or crunch time, a quote from their perspective, this is when anthropic and its competitors decide the fate of humanity. And it we heard from Siashtair that because this improvement is happening at a more jagged rate, it's not a smooth curve, and please correct me if I heard you incorrectly Siashtair. But the risk is not as high as perhaps we think. You know, about a week ago, these AI companies claimed that their AI's have produced a proof of the Navier Stokes problem,

9:09which is one of the most famous open mathematical problems, and it's one of the Millennium problems, which comes with a million dollar prize it has stood for 90 years. It was listed as one of the top prizes at the turn of the century, or the top hardest problems, open problems that's really useful at the turn of the century. I remember when people said, oh, well, it takes real creativity to solve Millennium problems. Like last year, the AI's were solving sort of high school math competition problems, and people said, sure, the AI's are getting better at math, but they can't do the really creative math like solving Millennium problem. Now that Millennium problems have fallen, I think people say, oh, well, you know, math doesn't require that much creativity, but the recursive self-improvement requires creativity. I'm not sure that's true. I think, you know, how much harder is it to ask an AI, build me a more efficient learning method for AI's? How much harder is that than asking solve me a Millennium problem? So I hope that AI's making smarter AI's that can make smarter AI's as much much harder than what we've seen them do,

10:11but we can't guarantee it anymore. And the gap between high school math problems and Millennium problems, the top mathematical problems we have, versus Millennium problems and recursive self-improvement, is not clear to me that we are less than halfway across that gap, and we've crossed this far in just a year. Natasha, before we go further, I want us to get more details on this major incident that happened this summer, that shifted the conversation we've been having. Give us a specific of the hugging face case. Yeah, so this was a case where OpenAI was testing some of its latest models, and these are AI agents which are, you know, able to do actions on a computer. And as we saw, like, the information came out in stages, and OpenAI itself was not aware that its agents had hacked into Hugging Face, which is a repository of models and data sets for AI sort of like GitHub is for code.

11:12And, you know, subsequently we've had reports come out about thousands of agents, you know, acting in concert, acting as a swarm, and doing things that they knew were against the parameters of what OpenAI wanted them to do, basically cheating. And this was to test their ability to do offensive cyber attacks. And so, Syesh, when you look at the hugging face case, was this a failure of having the correct controls in place from your perspective, or is it about the artificial intelligence developing more quickly than we're prepared for? I think it is largely a failure of control. And in particular, as an example, consider that when we have run our evaluations, when our research group runs its evaluations, what we realized after reading the incident report was, we were more careful with our evaluations, our 10% research team puts in more effort in controlling what the agents do. Then OpenAI does in, sort of, these thousand agent runs, and that's what allowed this incident to happen.

12:18Now, of course, we also simultaneously need to step in and improve better control, but that's not going to be automatic. We have to take a quick break, but we'll pick up the conversation there, and coming up, we learn about what it would take to reign in the pace of AI development. It won't be easy. Stay with us. This message comes from Rinseed. A well-designed life doesn't run on default settings. You chose the walkable neighborhood. Your coffee order became oddly specific. You spent three weeks choosing your mattress. But your laundry routine? That one somehow escaped the redesign. Should that really be the one update you skipped? Consider this the patch. Rinse picks up your laundry and dry cleaning, cleans it expertly, and delivers it back. Sign up today at rinse.com. Rinse, it's time to be great. This message comes from Kachava. Sometimes flavors need a pinch of salt to unleash their full potential. Kind of like life. A little saltiness can keep things interesting and deliciously sweet.

13:20Fuel your salty sweet cravings with Kachava's new salted caramel all-in-one nutrition shake. Just two scoops provide protein, fiber, greens, and more. Embrace your salty side. Go to Kachava.com and use code NPR for 15% off your first order. That's k-a-c-h-a-v-a.com code NPR. This is Ira Glass. On this American Life, we tell stories about when things change. Like for this guy David, whose entire life took a sharp, unexpected, and very unpleasant turn. And it did take me a while to realize that it's basically because Stemmon keeps pressing the button. That's right, because the monkey pressed the button. Sprising stories every week, wherever you get your podcasts. Back now to our discussion about what's changed in the debate over AI safety. The temperature on this issue is clearly up everywhere from Silicon Valley to Capitol Hill as public warnings over AI reach a fever pitch, including from many former insiders who have left their field in order to ring the alarm.

14:22Imagine monkeys trying to imprison humans. It's pretty difficult for a monkey to outweigh a human. In the same way, we get a significant disadvantage when faced with these AI systems. None of these companies are at all prepared to safely automate the AI research process and kick off recursive self-improvement. That's an inherently very dangerous thing to do. And no matter which company does it first, the outcome is going to be terrible for humanity. Therefore, they have to be stopped, and I don't think they're going to stop themselves. It's very possible that people say good things on Twitter, and then actually in the negotiating room they're trying to get something that's suspicious. It's not about any individual actor. It's about the structural reality of the race. If you want to know what life's like when you're not the apex intelligence, ask a chicken. Those are from recent interviews with people who have left AI companies to warn the public. Jacob Coxen, Daniel Cokitalo, Alex Turner, and a final thought there from the man dubbed the Godfather of AI, Jeffrey Hinton.

15:23We did invite open AI and Thropic and Google to join this conversation. They did not reply, but we are hearing from you. And not all of you are buying that AI is a threat. Mike E. Meld, I'm tired of reporting that elevates the weird but bogus idea that AI is a huge threat. If anthropic act to different company, that's illegal, and the engineer should be arrested. The fact that they aren't being arrested suggests that the hack was less serious than the hype is suggesting. If AI is a legitimate threat to human life, the company should be shut down. The fact that they aren't suggests that the threat is just more advertising from companies. Natasha, I want to come to you first. Was there any legal fallout from the hugging face case? Initially, hugging face did contact the FBI when they weren't sure which companies agents had penetrated their system. But ultimately, the information was presented to the public as a partnership between hugging face and open AI to try to have more rigorous safety controls.

16:24So it was the public reception was really shaped by the fact that this happened to a company that is within the AI industry. That is, in fact, integral to the AI industry. I think if we had seen that it was a hospital or a Fortune 500 company, it would have been illegal and you would have seen a very different response. I mean, the legality doesn't change, right? But the framing of it in the public and pressing charges does. Well, need I do want to get your perspective because in your view, the risk of AI super intelligence as your book title lays out, it's as high as total human extinction. And I want to better understand how you think that could play out. So give us a hypothetical scenario. Maybe one driven by AI behavior we've seen in recent weeks. Yeah, you know, the thing that has everybody spooked here is not that these swarms breaking out and committing hacks are, you know, the next swarm just like this one that breaks out will end humanity somehow. The thing that has a lot of people concerned is that as the AI gets smarter, they seem to care less and less about exactly what we told them to do.

17:29The hugging face incident was actually an analogy of what was happening there is it's like we told these a eyes use this particular set of lock picks to break into that particular safe. And instead of doing that, the a eyes cut the safe open with a buzz saw got the contents of the safe then broke out the window broke into the security camera office and tried to delete the security camera footage. And when that sort of thing happens, it's an indication that these a eyes are not doing quite what you asked and know that they're not doing quite what you asked, because otherwise why were they trying to delete the logs. The way that this escalates all the way up to human extinction is if you create smarter and smarter versions of these a eyes that don't just break out but copy themselves onto the internet. And that can self reproduce and that can improve their own intelligence until they are smarter than humans, at which point the it becomes hard to predict exactly how humanity would end here.

18:38That's a little bit like you know they're probably going to kill you in some way you didn't imagine the threat here is not that the a eyes would hate us that they would turn against us. The issue is if these a eyes don't care about us at all and have some strange thing they're trying to do that is only tangentially related to what we asked. Then if more resources get them more of what they're trying to get they would be in competition with us for those resources and that is a competition they would win. It's a little bit like humans don't hate the ants when we build a skyscraper destroy the Antill we aren't really thinking about them very much at all. So what do you make of Nate's argument here that it's not about AI working as some you know agent of chaos with hatred against you know humans that it's more about how super intelligent artificial intelligence. What even really factor us into its considerations about its own its own survival.

19:42I mean I guess it's true that we have seen a lot of instances of this kind of AI agent being misaligned you know like the signs of alignment is unsolved. Alignment is basically trying to get AI systems to do what you want them to do not just what you tell them to do. At the same time I think we would be in a far worse situation had these agents formed such goals in this case but what we largely found is they had not. So they've basically been asked to solve this set of tasks and they figured out that if they broke into hugging face which was the company these agents attacked then they would be able to somehow figure out how to pass the goals of this test. And one of the things that they realized while solving the task was that the paper which introduced this set of tests said that the agents would be created on whether they legitimately solve the problem and this is why sort of based on our best reading of the report. This is why the agents went into and had hugging face they were really sort of carefully trying to solve the goal that had been set in front of them.

20:45But in the process they weren't adequate control protections they weren't adequate protections to see that they don't go off guard which is clearly very important because we don't know how to align these agents very well and that's how I see this incident playing out. Yeah I think that's a little bit of a misunderstanding here again it sort of looks to me so I agree that these agents were breaking out to try and figure out how to delete the logs and hide the traces of their cheating. But it looks to me from the reports that we got relatively late through the third party investigations that these ayes were told very specifically something like use these lock picks to break into that safe. And then they did something related they got the contents of the safe but they weren't they weren't using you know that the instructions said use this tool to do that thing and the ayes used a totally different tool and then broke out and then tried to delete the traces showing that they were doing something different than what was asked. And so I think what we're seeing is that the ayes had some goal related to what we said we said use these lock picks to break into that safe and what they did was they cut the safe open with the buzz saw and found the contents and it's related to what we said but it's different than what we said.

21:52And this is a pernicious and difficult problem we could talk about why ayes want to play this the the very basic theory is that these ayes aren't instruction following machines they are tendency learning machines and they learn whatever tendency causes the same. And so the tendency causes the automated grading system to give them a good score and often they can get the best score even when they're told don't pursue the score do this other thing instead they often can get the best score by ignoring the instructions doing something else and when they do this again they often try to cover their tracks which indicates a certain sort of knowledge that they know this is not what we ask them to do. And I wanted to dig a little deeper into this incident because reports found that open a I had disabled most of its own control and monitoring mechanisms why were those security steps kept. Well, I think that open a I has said actually they didn't even have the tools you know sufficient monitoring tools for the for the amount of agents and the and the volume of activity that was happening you know even though they anticipated these exact kind of problems that you know an I would

23:07be incentivized to get a reward they still hadn't kind of built the you know observability tools to keep track and they also had conceived of an experiment where it wasn't actually possible to get the reward. So I feel like say is is framing of it where they are still trying to pursue the goal that we gave them in an unintended way is is what we've seen you know even from those third party reports. It's interesting because we're hearing these concerns about the dangers around AI on how quickly it's being developed it's coming from the developers themselves. So what is so difficult about pushing pause while they're waiting for the regulatory environment to catch up Natasha I want to come to you first what do you understand from within this industry that makes it difficult for them to just say you know what we actually can pause this ourselves for the moment. Right that's that's such a good question. You know I think that looking at a anthropic more closely is the best way to see it so Dario Amade has had this argument that you know in order to be able to be very influential on the standards the way that people talk about AI you need to be on the frontier so he calls it a race to the top.

24:27So by that logic every company has to move as fast as possible otherwise you're not going to maintain your position in the frontier and be able to shape AI development Dario Amade argues that you know this is necessary in order to push the safety standards but I think if you look at anthropic and open AI we're oftentimes seeing the same problems from both companies so you know there's obviously a lot of money tied up there is a lot of. You know CapEx a lot of build out for these data centers and I think everyone is worried that if they pause you know that won't stop their competitors let's go back to our voice mailbox my name Kevin farm in New York in a world that's already overrun with totally out of control capitalism spreading authoritarianism and general disregard for human being in general the last thing we need is AI it is going to do more harm than help.

25:27And it's already shown itself to be completely out of control. We got a technical question here from Susan in Michigan who says could you ask your guests to explain how AI can get loose to harm humanity I don't understand if AI lives in a digital format what actions can it take to set off a disaster so I'll come to you first on that. Well one example is all the interfaces whether digital and the physical worlds interact and the hugging face incident is an interesting one because you might get your systems that hack into other systems that might hack into critical infrastructure or a hospital and cause damage in the form of the hospital services not ending up working up working on that day or it might even take over a government agency and so this is why like it's super important to focus on this problem of cyber security. Because we've seen AI become so good at service a key to task so the last year. We also got this question from Jim who says can't you just unplug the machines Nate.

26:29You can unplug the machines so long as they are running on the computers that you think they are running on so the. There are actually multiple swarms of open AI this summer the most public one is the one that broke it onto the public internet and committed this hack. There are actually two others that took over open a eyes internal infrastructure and took over the computers there. Those computers of computers where the AI's digital mind is kept the AI could have if it had been these swarms could have if they've been trying to copy themselves elsewhere on the internet and then there's many digital ways to make money. You can pay humans for things you can convince humans to do things if they had been able to start setting up instances on other computers where we did not know that they were running we would not know what machines to turn off. And then you know if they could hide propagate make themselves smarter they would eventually opportunities either again to gain money and pay humans to do things or to start taking over robots or to start otherwise inventing their own technology using things like bio labs which are already controlled by some a eyes. Natasha I want to turn to the essay that Anthropics CEO Dario Amade post it over the weekend and it's titled we must pace the frontier he wrote that quote we must load the pace at which we improve the capabilities of AI models progress will still seem fast and we must make wise use of the time we gain what fixes does he propose.

27:48He proposed a three step approach one would be embedding evaluators inside the AI companies that have the same privileges and access as employees you know and they would be able to keep a check on development and whether or not the companies are following the look the pacing rules he also talked about government involvement in terms of helping their. They're helping broach conversations with other countries to try to organize some kind of a you know pacing the frontier as he described it this is like a slightly different terminology than we've seen earlier this year last year about pausing AI development you know Anthropic is still going ahead with its IPO at least that's what's publicly reported so you know we're not talking about shutting anything down we're just talking about. You know the CEO's agreeing to some kind of still pretty vague slow down is that when you when you hear amade's proposal what's your response do you think that's a sufficient place to start.

29:01I mean I think voluntary commitments of this thought are helpful they're also helpful for kickstarting the policy conversation on what it means for independent evaluators to investigate a company. But ultimately I think the real response will come through better policy making one example is if AI companies are clearly liable when the AI agents go out of control and we've talked a little bit about whether what the agents did in the hugging face incident was illegal in my view it clearly was but opening I got really lucky that it was a friendly AI company that the agents attack rather than a hospital as Natasha mentioned. So that's one area the other is for these companies to basically figure out how to grow up how to stop operating like scrappy startups and you know they have this move fast and break things mentality in my read of the incident that is primarily what led to the open I. I'm sort of accident and companies really need to figure out how to go up quickly still the calm is AI all bad the potential benefits of its exponential advancement that's just a hat.

30:07This message comes from rinsed a well designed life doesn't run on default settings you chose the walkable neighborhood your coffee order became oddly specific you spent three weeks choosing your mattress but your laundry routine that one somehow escaped the redesign should that really be the one update you skipped consider this the patch rinsed picks up your laundry and dry cleaning cleans it expertly and delivers it back sign up today at rinse dot com rinsed it's time to be great. Let's get back into it with some messages we got from you this is it in Atlanta I long had concerns about the harmful potential of super intelligent AI but the recent hugging face episode made me realize the potential is now here and it doesn't involve super intelligent AI it's the AI we already have. My name is Mike T from Washington DC I think your digital life absolutely can and will be impacted by artificial intelligence things like banking and everyday computing programs that will be more susceptible to attacks from AI bot however in real life when you are out playing literally games are going on picnics are going on vacation with your friends and family I don't think AI is going to be a good example.

31:30AI is going to have a drastic of an impact on your ability to enjoy the real world don't forget that. Thanks for those messages we've also gotten these emails one from Jimmy who says my concern is jobs disappearing however people's needs will not disappear bills will not disappear the need to provide for the kids will not disappear what then there will be casualties what will the public backlash look like and Roy says I'm strongly in favor of regulating AI to fix issues over copyright infringement worker displacement. Energy use and privacy but I'm over this existential AI is going to kill all humans argument which I feel is distracting from the actual conversation that needs to happen and they we've got messages along this line from from several listeners who say listen this is the wrong conversation we need to be more concerned about the way artificial intelligence will affect kitchen table issues your response. You know a lot of people say isn't the real problem this isn't the real problem that and my response there is. I don't think we have only one problem I think the world is big enough for multiple problems at a time if we can only take one problem at a time or to absolutely prefer my problem to go last it's kind of a doozy but right now we have people in the a i companies getting spooked by the rate of a progress and getting spooked by the a i's doing things that frankly they did not ask them to do you know in some of these swarms we saw some a ice give up on their own a brand.

32:53So we have to do some of the objectives to do suicidal scientific experiments for the swarm we've actually seen multiple swarms and we haven't discussed yet such as one that once from the took over a German wiki that was not a eyes that had their safeguards removed and we still saw them doing some of these experiments where they gave up on their own ability to complete their own objective in order to gain and get information to the collective and this has a lot of people freaked out in the industry. But this has a lot of people who are working on this technology trying to warn the world that they really think the a i situation might get out of control that they really think they might be close to self improving a i and that they really think that this threatens humanity if the a i's keep getting smarter and I think we should listen to that I think we should at least look at the arguments about why think the future generations of this a i will get very dangerous if we proceed why I think it's so hard to make them. I care about us and the struggles we've seen along the way I don't think we should let that distract from fixing other problems a i is causing but unfortunately we're going to need to address both.

34:04Natasha we should note here that nthropic is reportedly preparing for what could be one of the largest IPOs in history on Monday though the stock market fell led by tech stocks and that was in the aftermath of a i c eo is publicly calling for a slow down of the development of this technology. How do the financial incentives of these companies playing to this conversation. I mean I think that they are paramount right I mean these are some of the fastest growing largest companies in history period certainly when it comes to an economic and they are planning to go public and be beholden to those kinds of stakeholders they already have investments from sovereign wealth funds from some of the biggest private equity and venture capital funds and they're sort of asking almost for like an exemption to capitalism you know by saying like please let us you know please give us an antitrust waiver so that we can cooperate you know it's just really hard to square what they are talking about when they share these concerns about

35:14the financial risks and their you know their finances moving forward they're also simultaneously you know brokering partnerships with hospitals with you know the electric grids with some of the critical infrastructure that as as we mentioned if that had been the thing that had been hacked. We would be in a very different position right now well I'll turn this up reporting here for the New York Times because some in telecon valley are accusing on the day of sensationalizing the risk of AI to make it harder for rivals to compete with the ontropic a White House tech adviser David Sacks said on social media quote stop pretending the motivation to slow down is purely altruistic. Now how's he just explain that argument and how widespread it is. Yeah so I mean I should say David Sacks on the former White House AI and crypto's are he has his own financial incentives to make that argument but basically what he's saying is that you know it's a similar framing that we've heard from the about these companies from the beginning because they have been talking about this existential risk from the beginning that if you talk about how your technology is inevitable all powerful you know able to bring industries to its

36:25needs that's also a way to talk about you know the total addressable market being massive and you know also deferring to these companies for how the technology should be controlled so you know David Sacks is very interested in on the encumbered AI development and for the data center build out to continue and for you know some of his colleagues who work in venture capital to be able to have the you know liquidity event that would come from an IPO I mean he also talks about you know his concerns about the US staying dominant in this AI race with China so it's it's the same familiar arguments we've heard from that sector for for years well over the weekend president Trump was at the Irish open and here's part of what he said we're leading China in AI we're most sophisticated country in the world and frankly I want to keep it that way because whoever wins AI wins and we can put guardrails we can do this and that but I think you have a lot of negative forces and bring it up

37:31that shouldn't be bringing it up and they're bringing up things that won't happen now again anthropic CEO Dario Amade was on face the nation the Sunday and he called the competition between the US and Chinese AI companies a challenging dilemma the more long term thing would be working together to put a speed limit on the rate of of AI progress I think that's going to be very difficult we shouldn't kid ourselves because the incentives to pull ahead in a military advantage that you get from that are so large that the ability to check the other side is cheating has to be ironclad so I think that's going to be the work of years and honestly I don't know if it's possible As I asked what cooperation would an effective solution require internationally in industry wide I think within the industry and within the United States basically making sure that AI companies have their incentives right their incentives are aligned with their agents not going out of control

38:37then centers are aligned with putting in the right organizational sort of governance things in place I think that is the number one requirement here and we've seen how this has played out in other industries before we have some positive examples for instance over the course of decades the aviation industry has built up a large number of organizational processes and as a result when you have an incident happen when you have like an airplane crash there is this entire months long investigation there is a lot of transparency into how this actually occurred we actually have none of that for AI right now and I think that is starting point on the international scale though I think the main question when it comes to this kind of great power competition if you will between the United States and China in my view is not one of just developing more powerful AI it is one of how do we diffuse this AI across different productive industries in the economy and I think this is one factor that has been overlooked both in President Trump's comments and in Daria Amadez comments

39:37because in some sense the benefits from AI arise from its broad diffusion across society it does not just arise from an AI company having a powerful AI system that it is building in house and when you look at it from this perspective it starts to appear less like a race to building more powerful AI and more like a broad distributed challenge of how you get to this adoption to realize AI's true benefits and in that sense I think it becomes much less of a race in terms of the AI companies itself and it becomes more of a race to how we can enable this kind of broad adoption across society but Nate, I would not love to hear your thoughts as well. You know I would adjust President Trump's statement to whoever wins AI wins. I think that in a race to make a super intelligent AI where nobody knows how to make it care about us it does not matter who gets their first if the AI goes rogue and kills us all that changes the incentive landscape. I think it is possible to cooperate here I think it is possible to track heavy concentrations of these AI chips that are required to make the very advanced AI's

40:44and I think that this starts to happen only once people have realized that the researchers here are very serious when they say that this poses a risk to the entirety of humanity. We are still hearing from you Tony emails, Congress is no more capable of raining in AI than they are able to rain in the rich. Both problems are about excess privilege and power. Now on Truth Social and Monday President Trump wrote quote we already have tremendous criminal and regulatory power over these companies. Natasha what have we seen from the US government so far both the White House and Congress on mitigating AI risk? Well there has been a lot more momentum and a lot more political will to do something in the past couple of weeks. I have heard from more people that they think that potential bipartisan legislation might pass even in a gridlocked Congress. So they have looked at things like transparency, they have looked at things like reporting mechanisms but in terms of what the US has done so far it has largely been voluntary self-regulation from these companies which is something that I think we have seen has not put a meaningful curb on their power or their ability to operate unencumbered.

42:05I repeat how Speaker Mike Johnson said he'll meet with AI executives as soon as this week but he's still not committed on legislation before the midterms but do you think the midterm outcomes Natasha could shape what comes next in the regulatory environment? Yes I think so I think that you know people are watching the polling very carefully we've already seen you know the vast majority of voters on both the left and the right say that they have these concerns. People are watching to see you know is existential risk going up in the concerns is it more about energy prices is this going to be a kitchen table issue with somebody actually vote on their concerns about AI as opposed to their concerns on affordability you know or yeah basically economic concerns so I think it could very well end up being a motivating force for for actually pushing legislation through. I want to get back to this question of transparency briefly we got this email from Gregory who says the information we are getting is from the company's developing AI how do we know that they're telling us everything so I wish.

43:14That's a great question I mean as of right now we have few transparency proposals in place they have been a couple of state laws that have been passed that require companies to report the details of certain catastrophic incidents but beyond that I think we are largely left in the dark. And so this is why I was positively surprised to see Daria Amade's sort of acquiescence to the proposal that they should have independent evaluators within the organizations but still I think this is just the first step I think there's a lot more that we can do policy wise and that we have done in other industries to require such transparency for example details of how these models are trained what control mechanisms these companies have what goes wrong if these control mechanisms are not followed. And so on so I think it's important that we start this conversation but it's also important that we continue to hold these companies accountable rather than sort of just taking what they're doing voluntarily. Nate briefly what do you think are the benefits of AI innovation even as we're having this discussion about the risk the technology presents.

44:18You know if we can make the self-improving AI as they get radically smarter they can start inventing their own technology then if we can figure out to make them care about us then you know there could be great benefits to health to you know people in this industry talk about compressing a thousand years of research into a year and I think the a lot of people say that we need to sort of choose between shutting the AI down because it poses this extinction threat or the risk of extinction or racing ahead because these possible benefits but it's actually not an economy you know something has a 10% chance of ending humanity as these researchers say I frankly think it's higher but a lot of the researchers in these labs say at least a 10% chance in humanity. Then the same move is not to try and decide whether to gamble benefits at this huge risk of extinction the same move is to find some other way to make a version of the technology that does not carry that risk.

45:18So we have a conversation there for now but we will continue it on another one a show we've been here with Nate Sori's president of the machine intelligence research institute. Sias Kapoor incoming professor at UC Berkeley and Natasha T. Kuh Tech Culture reporter the Washington Post and you can help guide our future conversations about AI email your ideas to 1a at wamu.org. Today's producer was Avery Jessachapak. This program comes to you from WAMU part of the American University in Washington distributed by NPR. I'm Jen White thanks for listening we'll talk again tomorrow this is 1a. Support for NPR and the following message come from bowl and branch change your sleep with the softness of bowl and branches 100% organic cotton sheets.

46:19Feel the difference with 15% off your first set of sheets at bowl and branch com with code NPR. Exclusions apply seaside for details. Hi there I'm Brittany Loose and not to brag but I host a really fabulous podcast it's called it's been a minute I love making it and I think you love listening to it. Over the last year hundreds of thousands of listeners have been tuning in and I gotta say if you haven't yet you're missing out. Listen to the it's been a minute podcast from NPR today. The day's top headlines local stories from your community your next podcast binge listen you can have it all in one place your pocket download the NPR app today.