Is AI a threat to humanity?

0:00This BBC podcast is supported by ads outside the UK. What do Beyonce, Steven Spielberg and Cristiano Ronaldo as well as Paul McCartney, Mr Beast and Roger Federer all have in common? They've all joined one of the world's most exclusive clubs, becoming billionaires. But how did they manage to build such fast fortunes? Find out in good bad billionaire from the BBC World Service. The podcast that explores how some of the wealthiest people on the planet made their money. And asks whether they're good bad or just another billionaire. That's good bad billionaire. Listen now wherever you get your BBC podcasts.

1:00Is AI a threat to humanity? Why would anyone work there if they genuinely think they could be responsible for killing off the human race? A lot of the discussions of regulating AI happen at a 30,000-foot level that's separated from the actual challenges. It's world business report from the BBC World Service. I'm Vishanath Sripathma. On the way, a top researcher at Anthropics says AI poses a threat to humanity. So how worried should we be? Plus a blast from the past, Apple brings back the foldable phone. And the Indian food inspector has become an unlikely celebrity. So does artificial superintelligence really pose a risk greater than nuclear weapons? Is there a significant chance of a Chernobyl-sized catastrophe? Or many experts in AI say the fears are overblown. But on Wednesday, a top safety researcher at Anthropic,

2:03warned that the technology is advancing so quickly that as a greater than 10% chance, it could kill all humans within the next decade. Evan Huberger says he's concerned that AI systems could eventually become capable of improving themselves faster than humans can understand. Or control them. His comments were prompted by a post from Jacob Coxon, a researcher who recently left Anthropic after previously working at a rival firm, OpenAI. In a lengthy social media thread, he explained why he left, compelled to walk away. His treat has been voiced by our producer Neal. The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will coach their phrasing in the press to sound sensible. But I hear the same people express fears privately. No other human activity poses this level of danger. Dame Wendy Hall, who is a member of the UN's advisory body on AI, says she is shocked by the comments.

3:09Who is going to rein these companies in? Some of it could just be PR or marketing. You never know with them. You can't trust them. But how a human being can make a statement about our company has a 10% chance of killing off the human race. Why would you want to say that? Why would anyone work there if they genuinely think they could be responsible for killing off the human race? It's certainly a topic that's got lots of us talking. We speak to people in the north of England and ask them whether they were concerned about the increasing abilities of AI models. Some jobs like they are starting using AI and a seed of employer or some people. So it makes people lose their jobs. And about the tragedy biti, I use it most of the time. But I'm not dependent on it like 100%. It's a slippery slope because it depends about who essentially programs it. And the people that are programming it right now don't really fill me with confidence. AI companies are like open AI. Essentially they're just mass data farms that are just building profiles on people.

4:10And if you tie that into a citywide system, then you are building a mass surveillance system on the population. I'm a little bit worried about AI. I think already there's been plenty of examples of AI doing things that we didn't expect it to. Or AI sort of breaching boundaries that we kind of don't want it to yet. I think that it's probably not as scary as we think, but I think there's definitely the prospect of it being pretty scary. I absolutely love it. Yeah. I think it's great for supporting me teaching right now that they're finding things for cancer. So a mixed view, a mixed lot of opinions there. Professor Melanie Mitchell is a computer scientist at the Santa Fe Institute and joins us now. Welcome to the programme. Thank you for having me. You've previously poured cold water over these concerns. How are you feeling now because these big brains behind AI are worried? It's rather unearthing. Well, to be honest, this is nothing new. This kind of thing, these exact same statements, the 10% risk of human extinction.

5:18And so on have been being predicted for decades now. And we haven't certainly haven't seen it happen. We haven't seen that AI systems are getting closer and closer to killing off all of humanity, which would be quite a feat to be honest. But there are certain risks that are coming to the fore now that I think do need to be addressed. And what are these risks? Well, there's many risks that have been around for a long time, such as the risk of AI systems sort of polluting the internet with misinformation and disinformation and many more. But now we see that these systems that are being created now by companies like OpenAI and Anthropic, these so-called agents that actually go out and do things on the internet. They can do very sophisticated kinds of hacking and attacking other companies.

6:18And the way that they're being trained now does present quite a big risk, I think, to our cyber security. And do you think that there is enough regulation rules around it to prevent real threats? Absolutely not. There is very weak regulation in many parts of the world in the US. There's almost no regulation, which is these companies are left to quote unquote regulate themselves. And it's a very dangerous situation. Okay, well Melanie, please stay with us. Well, these resignation and comments at Anthropic prompted Darren Jones, who was chief secretary to the Treasury here in the UK and later chief secretary to secure Starmut to write an open letter to Prime Minister Andy Burnham and the Secretary General of the UN and OECD calling for a new multi-national treaty for the safe development of AI.

7:19And he joins us now. Welcome Darren. Hi there, great to be with you. So what would actually be in this treaty? What rules would you put on governments and AI companies that they don't face today? Well, I think what you're trying to achieve is that on this narrow question of super intelligence, the idea that these AI agents can communicate with each other and decide what they want to do without human instruction or insight that we want to create a safe regulated environment for these technologies to be developed with a human understanding about what is happening. And what those risks are and whether they should then be deployed in the broader marketplace or not. Well, I don't think we want we don't want to complete some other AI and its general applications, which I think has lots of potential benefits for the economy and for public services, but on this particular issue of super intelligence, I think we need to be looking at it more closely. Is that balance a sort of almost a sort of work of fantasy because you know we want Britain or we keep hearing from the government that Britain needs to be this AI superpower and attract billions in investment and yet also being told that we need more regulation.

8:27Is there a there's a trade off there, isn't there? I don't think so because to be honest, the day to day applications of what some of these AI agents can do, whether it's helping you complete your online shopping or fill in some forms, I mean, that's welcome, we should we should go ahead and do that. But I think you know it's a pretty traditional understanding that if there's a new technology that poses safety risk to customers or to consumers, the functions of the state should have a look at it on behalf of the public now. The debate today is ranged from extinction of the human race through to people accusing anthropic of just marketing hype in advance of their IPO as a legislator and as a former minister, like we should just check what we think it is. And if it's more towards the extension risk, then clearly we need to take more action. And is there a problem though if we do accept that that we are going to have slower development as a result. Well, firstly, I think it needs to be done on a multinational basis. There's no point in Britain doing this alone. And that's why I've called for multinational action, both at government and parliamentary level today.

9:33But I think you know the companies themselves are asking government intervene to slow down the pace of development at the frontier, which is essentially what I think needs to happen in respect to this question of super intelligence. Thank you. And we're just going to pause to hear from Chris Miller, who's a professor of international history at Tarsie University, whose research focuses on tech politics, geopolitics. And he's been looking at this idea of multi national regulation around AI. It's already clear that like any technology, AI will have lots of positive uses, but also some negative applications. We've seen the risks in terms of cybersecurity over the past couple of months. And one of the key challenges is that governments are inevitably going to try to use AI to compromise their adversaries cybersecurity in a competitive fashion. And the US and China will be two of leaders in this effort. And so on the one hand, governments don't want their economies to be attacked by more powerful cyber capabilities.

10:33And every government is going to want more capable cyber offense tools for them to use against their adversaries. And this creates a core dilemma that governments are still trying to grapple with. And it's interesting what's happening in Europe, because Europe is criticized for being a little bit behind. We've got Mistral in France, and there is a lot of praise from American companies that France has sort of almost taken the lead there and developed its own large language model. How significant is Mistral though? Do you think it's going to be able to compete with the likes of the US and China? Mistral is clearly a very capable team. They've produced some impressive models. But what we have seen for the most part is that when European companies want to use AI, they disproportionately turn to the large US companies that can provide it at scale. So I think there will be plenty of space in the future for open source model providers like Mistral for certain use cases. But right now the market is telling us that there's also a lot of demand for the highest end AI tools, which today are produced by companies like open AI and Anthropic, almost exclusively US firms.

11:41And right now that's what the market is certainly willing to pay for. And so do you think that Europe does need to sell catch up and do less in terms of regulation? Well I think it certainly is the case that Europe aired in regulating AI before it had any AI companies. And the people in Brussels who were proud to have taken that stance, I think today look a little bit foolish because it's hard to regulate at the global level when none of the key companies are in your own jurisdiction. I think most of the economic benefits of AI will come from deploying it at scale, using it to make medicine better, using it to make logistics more efficient, using it for a whole range of purposes. And I worry that most governments spend too much time talking about regulating a relative amount of time they talk about actually harvesting the benefits of AI, which I think could be substantial. And so I know you alluded to this earlier, but do you think then with that in mind that given that they're all sort of invested in developing AI at a similar pace, would there be a situation where these sort of big markets come together and come up with some sort of regulatory frail.

12:45And so that's what we're talking about is that kind of a sort of regulatory framework to sort of work with it. And always that kind of pointness because we keep hearing about how AI is really dynamic and is changing constantly and you can't really regulate it because it keeps changing. A lot of the discussions of regulating AI happen at a 30,000 foot level that's separated from the actual challenges. I think if you ask, do we want AI autonomously undertaking cyber attacks? The answer is almost certainly not. AI that supports cybersecurity rather than undermines it. If you ask, do you want AI devising dangerous viruses? The answer is almost certainly not. The discussion about regulation gets much more tangible when you talk about specific potential harms. The problem is most political leaders are still talking at an abstract level. So that was Chris Miller, professor of international history at Tufts University. Listening to that was Darren Jones. Chris there was saying how governments are thinking in an abstract level. Do you think China or the US will play ball when it comes to this new treaty, for example, that you're proposing?

13:49Well, my hope is that on this narrow question of super intelligence where humans lose control of what AI agents are doing, you know, there's a shared interest there, whether you're Chinese, American or from whichever political party or country you're from. There's clearly going to be continued competition, both economically and in terms of military and national security application, which I agree we're not going to be able to stop. But I hope you would be able to appeal to countries around the world on this narrow question of super intelligence and bring people together to find a way forward. And Melanie, it's not keeping you up at night, but perhaps the idea of the lack of rules might be a little bit concerning. I do find it concerning, you know, just to push back on the notion of super intelligence. I don't think that is a well defined term. And I don't think it's specific enough as in the words of your previous guest to regulate.

14:56I think that it in some ways, this attribution of quote unquote super intelligence is in a way, taking the, taking the liability off the hands of the companies and putting it on the AIS systems themselves. And that I think is really backwards. Well, Professor Melanie Mitchell and Darren Jones, thank you very much for joining us. Let's check in with our financial markets now and a season to it is portfolio manager and exchange capital resources in Chicago high season. Hello, always great to have you on quite a few market stories around today. But let's stick with this theme of AI now AI infrastructure, particularly makes up a lot of companies in the US indices. So is it fair to say that investors almost need them to win at this point despite all these warnings that we're hearing? Well, I think the investor mindset is that the gains in the market have been driven by AI over the last 18 months.

16:02Certainly AI has been the story for investors since Nvidia first announced their amazing number of chip sales and the possibility of what could happen with their GPUs, their graphic processing units, which was the beginning announcement of all of the AI development that we've seen. Investors have just learned for management teams that literally hundreds of billions of dollars have been sent spent in this AI infrastructure build out. So there's very much an investor sentiment that it has to work. The market is intertwined in this development. We have two very big companies andthropic and open AI in the US waiting to come public investors are looking towards those IPOs thinking this is going to be the next big thing. And again, this ecosystem now has got so many people involved that it's almost difficult, I think, for investors to envision it failing. Yes. And so, you know, there's a bit of an arms race that we were leading to earlier on every country wants a skin in the game.

17:04It's quite how it's aligned at this point is not what you're saying. It is hard to unwind at this point. There's so much capital that's been deployed. There's so much development that has been done and that is ongoing. And there's no real governing body to pull it back across the globe. And so I think that also is, as you said, the arms race. And so, no one wants to compete to have that best model, whether it be between companies or between countries. And that is what makes it so difficult to stop at this point. And it's just briefly the oil price made headlines day. It's hit new highs. Oil has hit new highs were up over $100 a barrel. This is a big reversal from where we were just a few weeks ago when we saw oil comfortably in the mid 70s. So, obviously a reaction by the markets to the increased tensions and exchanges of fire between the US and Iran and the various events going on in the Middle East as we've seen in the last couple of days.

18:06Tensions are heightened. Investors are reacting to that. And as a result, oil prices are going up. Yeah. So, we are seeing that on the reverberations on the market. Thank you very much for joining us today. Susan Schmidt, portfolio manager at Exchange Capital Resources in Chicago. You're with World Business Report from the BBC World Service. What do Beyonce, Steven Spielberg and Cristiano Ronaldo, as well as Paul McCartney, Mr Beast and Roger Federer all have in common? They've all joined one of the world's most exclusive clubs, becoming billionaires. But how did they manage to build such fast fortunes? You find out in good bad billionaire from the BBC World Service. The podcast that explores how some of the wealthiest people on the planet made their money. And asks whether they're good, bad or just another billionaire. That's good, bad billionaire. Listen now wherever you get your BBC podcasts.

19:15On Apple podcasts. You can also enjoy a range of our podcasts, add free, with an Amazon Music subscription. And we're getting to stay in the United States because John Ternis has barely settled into the top job at Apple. But he's already overseeing one of the biggest changes to the iPhone in its near 20-year history. On stage at the company's annual product launch, he unveiled Apple's first foldable iPhone. The new device called the Duo has been years in the development and also marks a major redesign of the firm's flagship product. Hello and welcome to Apple Park. I am so thrilled to be here today. Before we get going, I want to take a moment to express what an extraordinary privilege it is to represent the remarkable teams here at Apple. It seems very happy to be there, doesn't it? John Ternis speaking at that event. Joining us live now is Lily Jamali. Hi, Lily. Is this a bit of a gamble for Apple because the smartphone, the iPhone, is quite iconic and people are quite reliant on it.

20:19It is. I think that Apple is framing this as the logical place to take the iPhone. They acknowledge it as the biggest change to the phone since it was released almost two decades ago by legendary CEO and Apple co-founder Steve Jobs. But I think they need a new form factor as they say and they have that. They are still making standard iPhones and we heard about the iPhone 18 Pro and Pro Max today. But I think that this is an opportunity to really move things forward and try to remind people that Apple is a hub of innovation at a time when some people criticize them for not having had a blockbuster new product in many years. So, Lily, I'm looking at the phone. It kind of reminds me of those early computer phones that my dad had like in the 90s. It's a bit of a throwback. I hadn't thought about it in those terms actually. I love hearing feedback. I've been in the confines of Apple Park for the last several hours hearing real voices and real opinions is really something.

21:27But, you know, I actually I have to say, you know, as I was hearing John turn us talk about it. He was talking about how the engineering of the hinge was so important to create seamless opening and closing and sort of so on and so forth. But when I actually saw the phone, it actually does look seamless when you open it up. It looks like an iPad. If you were just walking by it and not knowing that this was an entirely new product that Apple was releasing. It's got a pretty sizable screen when it's open 7.6 inches. I think, you know, usages could be watching video, of course, you could read books on it. And it also kind of has this dual screen. So you can be doing multiple things at once. And it's it is actually the finished iPhone ever when it is open. So there's a lot to like. But on the other hand, it's I think you're going to see some compromises made with respect to camera with respect to battery. I talked to some analysts here. They say those compromises are fairly minimal. But that is one of the things about foldable phones. And of course, there is the high price tag which we have to talk about.

22:36It's not a cheap phone is it really it's quite expensive and look, can you imagine the insurance on it? It's not because these days, you know, Apple car plus is something that Apple is pushing quite a bit because their phones are expensive and you're a bit vulnerable if you walk around with one of them. They are quite expensive. So in 1999 to start in terms of US dollars. But if you're in the UK, you actually will be paying 1900 and 99 pounds for the phone. And then prices can rise quite a lot. And hundreds of dollars or pounds or other currencies because this will be available in multiple countries next month, depending on how much storage you opt to buy for the phone. So it could be more than $3,000 ultimately when all is said and done. That is a lot of money for a category that has been has sort of eluded consumer interest. Samsung has tried Huawei has tried as have others and foldables remain just 2% of the smartphone market. And it will leave you a beef and now just one word on John turnis was a quite excitable. It was his first first little event, right?

23:43This has been the most coordinated corporate handoff ever seen. He seemed steady. It was all videotape by and large. So, you know, I'm not sure how many takes he had to do, but it felt very much like a seamless handoff. Tim Cook making a brief appearance in the video and very gracefully passing the baton. That's very kind of. Thank you very much. Really to Marley there at the Apple new product launch event. Well, we can head over to India now because a food safety crackdown in India's financial capital Mumbai has gone viral thousands of restaurant raids and photos of filthy catches and shed widely online. The BBC's Ujshana Shukla met the man behind the enforcement drive the state's new food safety regulator is become something of a celebrity in India but a controversial one too. Do you have filter coffee? Yes, yes. Can I get one filter coffee and a plate of idli sambar please? Yes, yes. Thank you.

24:44I've been craving filter coffee all morning so I've popped into this restaurant here at Mumbai office and this place looks different from the last time I was here. There's a fresh coat of paint on the walls. There are new bins. The open kitchen counters that I can actually see from here are being wiped down after every task. And the staff here also seems to be double checking everything like nobody wants to end up in the next viral video. And that's not really a joke because the recent photos and videos coming out of the food safety raids across my restaurant that's the state Mumbai sits in have been hard to ignore cockroaches and kitchen floors spider webs on the walls rotting vegetables. And behind it all is Tukaram Mundi the state's new food safety commissioner since he's took over in May his teams have rated more than 4,000 food businesses. That's over 40 raids a day. 300 licenses have been suspended so far for violations. When I joined I never anticipated this will be like it. We turned out that way.

25:45I met my office in the city's business district. What have been some of the common violations there? I did not expect these misviolations. In frustration we have seen cockroaches and rats. We have seen even some of the places absolute and hygienic condition in the kitchen where we just cannot think it's a filthy condition. Wherever we found these violations we have gone for food suspension and we are taken for the action. And it's across from multinational food chains to a hundred year old dairy to the iconic and elite clubs all have had their kitchen shut down. Mundi's office says they've even received 16,000 public complaints. This has made the new food commissioner popular but also controversial. Some businesses say they were shut without warning and have taken legal action. Coats have at times find Mundi's department called their actions overboard even reversed some decisions. There's been some amount of criticism as well with some establishment saying that the actions are more strict than needed.

26:49Had this action been taken earlier and had the food business operators or establishment taken complied by this regulation I would not be needed to talk those actions. The actions which we are taking is as per the law. It is in the public health interest and it will continue so that we eat whatever we eat and whatever purchase is a separate food. But Mundi has tapped into the shared public frustration over food safety in India where government data shows one in five food samples tested were found unsafe, substandard or mislabeled. And that's why Mundi's hard enforcement style has made him popular. There are selfies, fan crowds close to four million Instagram followers. Experts however say that why this has raised awareness one officer can't sustain it all. Building a system that outlasts him is the real challenge. But now the change is reflecting even in street stalls like the one selling sandwiches here that have stopped using old newspapers where the ink can seep into food.

27:49After Mundi abandoned it among many other unsafe items. Today I actually got my sandwich wrapped in flame grease spoof paper. I was going to have a snack from the canteen after the show. I think that report might have hit me off. Anyway, great reporting there by Archener Shukler. That's it for my World Business Report. Thanks for listening. You can follow me on my socials. I'm at BBC Vashalla SP. We'd love to hear from you. Thanks for listening. What do Beyonce, Steven Spielberg and Cristiano Ronaldo as well as Paul McCartney, Mr Beast and Roger Federer all have in common? They've all joined one of the world's most exclusive clubs becoming billionaires. But how did they manage to build such fast fortunes? Find out in good bad billionaire from the BBC World Service. The podcast that explores how some of the wealthiest people on the planet made their money.

28:52And asks whether they're good bad or just another billionaire. That's good bad billionaire. Listen now wherever you get your BBC podcasts. You can also enjoy a range of our podcasts, add free with an Amazon Music subscription.