Why are AI leaders calling for a slow down?

0:00During Lowe's pro savings days, my Lowe's pro rewards members can save up to 20% when they buy more job site materials. Save 20% when you buy 30 or more select bats and rolls of fiberglass insulation. 24 more cartons or pales of joint compound tape and bead. Or when you spend $200 or more on free and painting and job site essentials, members give more at Lowe's. Valet the 195, while supplies last. Selection varies by location, program subject to terms and conditions, C-Lows dot com slash terms for details, subject to change. This is a CBC podcast. Hi everyone, I'm Jamie Plesson. It has been a remarkable week in the artificial intelligence world. Last Tuesday, a researcher resigned from Anthropic saying AI companies are, quote, racing straight to self improving super intelligence and gambling with our lives.

1:01A current Anthropic employee agreed saying he gives AI more than a 10% chance of causing human extinction within the next 10 years. Then on Saturday, Anthropic CEO Dario Amade published an essay calling to slow the development of AI so that safety measures can keep pace with the technology. His competitor Sam Altman of Open AI and Elon Musk of SpaceX AI endorsed the idea. This all led to a mix of alarm and calls for regulation on the one hand, as well as skepticism and a search for ulterior motives on the other. Gary Marcus is a frequent skeptic of the AI industry's claims, but he wrote a post on Sunday, which he called a partial endorsement of Dario Amade's essay on his substack, Marcus on AI. So he's back on the show to talk about it. Gary is a cognitive scientist, an AI expert and author of books, including Taming Silicon Valley. Gary, it is really great to have you back on front burner.

2:03Thank you for making the time. I'm pleasure to be here. So you are, I know, usually pretty skeptical of AI companies, but you said Dario Amade's essay, quote, may prove to be one of the most consequential essays of the year. If not the decade, why did you write that? Why do you believe that? Well, I was I so nice. Um, I wrote that because a lot of people are getting in line with what Dario said. So, you know, whether it's actually consequential, I guess remains to be seen in a certain sense, but, you know, as you already noted, he got Sam Altman and Elon Musk in line. Altman posting, I agree with Dario that we need to pace the frontier. Elon Musk and top executives from Microsoft and Google also saying they agree. And he also got a counter reaction from Donald Trump who said, we don't need any of this. We're fine. In a new post on social media, the president said, quote, the only control or guardrails that AI needs is a strong and smart high IQ president. And the USA has that.

3:04He went on to call the warnings, quote, a sick conspiracy going on against AI and data centers. And the only one that is happy about it is China. And so, you know, we're talking about and everybody is talking about that already makes a consequential. If it actually leads to a slowdown, which in my that would be a real change in how we're developing AI right now. And I think, you know, there's a real chance that that will happen. We are in this mailstrom vortex right now where it's very hard to predict even what's going to happen an hour from now. I mean, I was talking about this stuff yesterday. And then the BBC said, we need you to talk about King Charles and AI. And I'm like, King Charles and AI, what is that? It's calling like some sort of beating. I think he's calling a meeting. Exactly. Everybody's calling me like, and you know, Trump and she are about to meet. I just read about that and economists this morning. Like, there is a lot happening, but Dario's having written that and gotten some people on board certainly has potential for consequence in the world. I want to talk to you about what it is concretely that Dario's calling for.

4:08But I first want to break down his concerns with you. He writes about something called recursive self-improvement, which is AI's ability to improve itself. He says this is starting to happen across the industry. And that if unchecked, humans won't be able to understand or control it. Writing of his fear that rogue AI agents could be capable of taking over the entire internet within the next six to 12 months. If we build in the right way, I think the probability of something bad happening is very low. If we build in the wrong way, the probability of something bad happening is very high. Do you think that that is something that we should be worried about? Yes and no. And it partly depends on the time frame that you're talking about. So there are a lot of terms in AI that are loosely defined. And recursive superintelligence or terms like that are pretty loosely defined. The broadest notion of it is you have a machine that improves itself. And that can be very narrow or very broad.

5:11On the narrow side, like we have these things called neural networks now. And they get trained on data. And you could have one of them figure out how to let's say train on 99% of the data instead of 100%. It could optimize what it's doing. That would not fundamentally change how the machines work, but it might do the work that a human engineer might do. Maybe it does a little bit faster. I would call that improvements around the edges. What I think people have in mind is something more dramatic, where the AI says, hey, I don't really work very well. I could do this thing that turns out to be a major breakthrough and it would be game-changing. That's the kind of thing that leading scientists at least occasionally do. Is they come up with a fundamental change in a paradigm. They come up with a whole notion of deep learning or symbolic AI or Einstein figures out relativity or that kind of thing. Right now, AI is not doing that. It's not doing game-changing discovery.

6:13There's always a bait and switch I find in AI. Like with the term AGI, artificial general intelligence, you have people like artificial general intelligence is going to kill us all. And then you show, well, the actual thing that we have right now is flawed and to hallucinate them, whatever. And they say, but people do too. And so the bait is this version of artificial general intelligence that's going to change the world. And the switch is, well, we're going to just label whatever we have to be artificial general intelligence. Those two things are different. And that's happening a little bit with recursive self-improvement. It's like there's some trivial sense in which that might already be true. And then there's the stuff that Dario is talking about that would be like game-changing transformational change the world in this way or that. And I don't think we're actually particularly close to that. And what was you have to see to change your mind on that? I would want to see to change my mind on that some real scientific discovery that is beyond just kind of brute-forcing like we're seeing in math. And I would want to see it in open-ended domains,

7:13which is really what science is in AI is. So the one place where we're seeing something like that is with math. But with math, you can use these theorem-proofers and things like that that are not actually part of the neural network, although that's a separate conversation. And they can formally verify certain things. And the problem with most science is it isn't like that. Most of the interesting things in science, you can't just stick it in an equation and kind of check your formal proof. And that's where you actually need the creativity and the insight. And a lot of it is like redefining things, building new paradigms and so forth. You're not really seeing that in the math. In fact, in the math, the write-ups of these things are like boring and uninteresting. It's just like a long chain of symbols. Mathematicians like Terence Dow have said, you know, this is not actually even what we really want in math. And he has argued that what we're seeing so far is this kind of brute-forced stuff that doesn't have mathematical insight. You know, what would change my mind would be insight from these systems?

8:14And I haven't seen it. I'm not saying it's impossible, right? I want to be clear that I think that artificial general intelligence is possible. I think recursive self-improvement is possible. I think insight from AI is possible. But it's certainly not a robust property of current AI. What we're missing, I think, in the kind of public conversation right now is a kind of grounding in reality. So like this guy who's everywhere now, you ubiquitous, the former employee of OpenAI and Anthropic. Jacob Kotsen who had this tweet that had 150 million views. Who's viral post- ominously warned, the people building AI earnestly believe it could kill us all by the end of the decade. Thinking about it in terms of an alien mind that we don't fully understand is pretty terrifying, even without any concrete scenario. People write conspiracy theories about it, whatever. And he's on every television show. He's about to do something with Bernie Sanders and Steve Banner. We could talk about that. He's running around saying that it is basically quite likely

9:18that in the next five years, we might see the extinction of the human species. Well, that is absurd and ridiculous. That is not going to happen. I mean, thinking about COVID killed a tenth of a percent of the human species. One of the worst threats that we ever dealt with. But we also did deal with it. And we did, you know, caught our losses. I won't say to a great number, but nothing like actual extinction. And he doesn't have any like systematic thought through scenario. Where you could evaluate and say, well, the odds of this thing happening, whatever, there was a book last year that had a similar kind of mania around it, called, if anybody builds it, everybody dies by Ali Eserukowski and Nate Sures. Some of their assumptions are reasonable. But when you look at the actual scenarios by which everybody dies, it's all kind of science fiction fantasy land. And there's no reason to think that these particular things are going to happen soon. One of the things they tend to gloss over is that humans can fight back. So they have this idea that if AI is smarter than us, we won't be able to do anything. But that doesn't follow.

10:18I mean, bears aren't smarter than us, but they'll fight back if we get enough fight with them. There's just a lot of silliness in those arguments. I just want to put to you the existential risk argument that I hear, which I've also heard people refer to as like the paperclip problem. We ask an AI to maximize paperclip production, and it starts like tearing the world apart to make paperclips and turns on humanity when we get in its way. And this is like tongue and cheek, right? But you apply this to climate change. We tell AI to fix climate change, and it realizes that we are the source of climate change and therefore have to be eliminated. And just how do you respond to this paperclip problem scenario? So there's a couple of things. One is there's a core intuition there that is right. It's like sorcerers apprentice. We tell a system to do something.

11:18It doesn't understand what our instructions are. It winds up doing something else. There's a term in the field called reward hacking. And this is a real problem. So the open AI hugging and face incident was at least a version of that to some extent. Chad G.P.T. creator, OpenAI, says two of its models escaped controls and successfully hacked into another AI company. OpenAI says they are the way they describe it is responding accordingly. It doesn't necessarily lead to the extinction of the human species for a bunch of different reasons. One is if we see a system like that, we can shut it down at least for now. We get ourselves in a position we can't shut it down. And that's a problem. Two is if we really built artificial general intelligence, we ought to be able to build it with a sense to follow instructions. That particular example, the famous paperclip example, assumes without saying so, that the machine is too stupid to understand instructions like don't kill humans. And we should not build AI that is too stupid

12:19to follow instructions. Now here's the sad part. That's exactly what we're doing is building AI that is too stupid to follow basic instructions. So the hugging face incident, they should have told the system, I don't know if they did or not, don't hack into other systems. And what we are finding is with particular technology, which I would call general purpose agents built on large language models, they're not very good at instructions and they're causing some harm. What we should do right now, in fact, is the equivalent of a factory recall. We should say, hey, these things are causing problems. We don't really have a good handle on that. Let's hold them until we do have a better handle. And we could perfectly well do that and not go extinct. On the other hand, we might be stupid and not do that. And there's a kind of social political issue there. But at the moment, we could easily just shut those things down if we wanted to. And then they would pose no danger at all. We just shouldn't give those things internet access. So you'd have to get to a place of a lot of stupidity.

13:19And I am seeing some stupidity in order to get to something like the paperclip scenario actually being realized. The hugging face incident that you refer to, this is another AI company that was hacked by open AI's agents, right? And just my understanding is they have no idea why those agents even did that. Like could they have even anticipated giving it the instructions to not go hack hugging face? Well, you'd want to give a broader instruction. Like don't steal credentials. And they didn't even do that. There's a lot of things that went wrong there. I have a source who indicates they probably they actually put some pressure on the system to do this. Number one, and we would like to see logs and stuff like that that open AI is not revealed. So we don't actually know those circumstances. Number two, we know that open AI actually relax some of their own guardrails, what they called in their report production classifiers. They turn those off.

14:19So in some sense, it was a laboratory demonstration. They were like, what can we go this system into doing? It was not something organic in the wild. They actually made it easier for the thing to get out. Number three, they didn't put it in a proper state of the art sandbox to restrict it from the internet, which was, I think, a mistake. Number four, they didn't monitor well enough. Basically, open AI was not following industry standard cybersecurity measures. Had they done so, this wouldn't have happened. In fact, they acknowledged, but this gets worse in a second. They acknowledged that if they had monitored more carefully, they could have caught it earlier. So there's standard stuff about monitoring and they weren't doing it right. Now, the way that this gets worse is that everybody in the field knows right now that there is a fallible, not great, but better than nothing technology called chain of thought monitoring in which you force the systems basically to articulate out loud what they're, it's hard to say this not anthropomorphically,

15:20what they are, quote, thinking, close, quote, right? Now, it's fallible because what they say they're thinking doesn't always reflect what they all actually do, but it's correlated with that. And what open AI just did is they released a new model, GPT-6 Astra, that reduces how monitorable their systems are. In my opinion, that's about the worst thing we could do in the world right now and it's also emblematic of a really flawed system. So it's the worst thing we could do because we're taking the only thing we have right now, where, okay, I exaggerated a little, one of the very few things that we have right now and they've undermined it, that's not good. But it are also more than that, reveals how bad our system is right now and why we do need more regulation and so forth. That system was vetted by the US government and they let it through. That's problematic. What it tells me is that the US government doesn't know enough about AI. Now, there's a second point I want to make around this, which is my biggest fear around all of this, is that if we don't have good government procedures

16:22in place, it ultimately lands on the companies to decide what they should release. And they have enormous economic incentive to release essentially everything. I kept warning about this as a hypothetical for years, possibly even on this show. I said, look, what if they come out with some kind of thing that allows people to make bio weapons a lot more easily than they can today? That could be really bad. And I said, I wouldn't want Sam Altman to make that decision because he has a complex set of motives. Well, so far we haven't reached that point, but we did just reach it with GPT-6 Astra, right? There was a system that probably should not have been released. It should have been kept on the shelf for a while. And this will lead to some other conversation too. Instead of keeping it on the shelf, they released it. We know there are other incidents. We know it was involved in the Hogging Phase Incident. And he made that decision, I think, I am speculating, because of his economic incentives, right? His economic incentive is he's running out of money

17:22and he needs the IPO. And so he wants to have big publicity things. We could talk about that as well. And so having a new model gives him publicity. I doubt that he weighed the risk all that much, or certainly not as much as I would. And so it's a really good demonstration of the fact that the CEOs of these companies have way too much power and the government has failed to restrain them. That doesn't mean we're all going to die, but it does mean we could see, for example, a lot of really bad cyber acts. And new power grids taken out and stuff like that. You're saying a lot of stuff that's making me feel uneasy here, Gary. I'm here to calibrate your unease, just to be clear about that, right? I'm telling you, you don't have to worry about extinction, but there are other things to worry about. So you should still feel some unease. During lows, pro savings days, save more on the products you count on to get the job done right.

18:24Get a free two pack of DeWalt 20 Volt Max XR 8 Amp Hour Batteries when you buy a Selex DeWalt 20 Volt Max Power Tool. Plus, save up to 35% on Selex Major appliances. The right products, the right deals, ready when you need them. Our best lineup is here in Loads. Valetue 925, while supplies last. Selection varies by location. Hi, this is Dave Roberts. Heading into summer without a medical emergency kit, that's a risk. Summer colds can linger and getting sick on vacation can derail everything. That's why everyone needs a medical emergency kit. It includes doctor prescribed medications to treat common and serious illnesses. So if you wake up sick, at home or on the road, you already have what you need. It's like having an urgent care and pharmacy at home. Say $45 with code blue at urgentcarekit.com slash blue urgentcarekit.com slash blue. On that point, another big concern is the hacking into weapons or the creation of some kind of bio weapon and anthropic releases report.

19:27On Thursday last week, saying that they intervened in several cases this year where scientists might have been using their tools to develop biological weapons. They say it's hard to tell for sure, so they aired on the side of caution. Just to be clear here, do you see a world in which AI could be used in this way and fairly soon? To create bio weapons and so forth. Or hack into a drone and start shooting people that just starts shooting people. Yes, I think I see all of that. I would like to interject something first, which is that this is one more example where the AI companies are behaving as unelected governments. Essentially, anthropic is deciding what research can and cannot be conducted. There is as far as I know, no outside scientific advisory panel. There is no government oversight of what they're doing. That in itself is yet another worry. It's akin to worrying about totalitarian governments, but it's like totalitarian companies. They may actually have made the right call here, but I wish that some outside scientists had some say in it.

20:34Leave it at that. Is it possible that AI can hack into drones and cause mayhem? Yes. The magnitude of it before we can stop it is unclear, but I think that that is possible. Is it possible that AI could be used to synthesize bio weapons that we haven't dealt with before? The answer to that is clearly yes. There are things there to be uneasy about. They're not going to cause literal extinction, but they could cause a lot of casualties. I want to ask you or I want to talk to you concretely about what you would like to see here. Dario is not calling to stop AI progress. He's calling to pace it, to slow it down, to the point where he says safety protocols can keep up with the speed of development. He's suggesting a couple of things. First is granting broad access to third party evaluators in order to oversee safety practices, report incidents. Then I think the next step after that is in some form the companies need to work together to set standards about safety standards for release, perhaps something about the pace of release.

21:45But in order to do that, there needs to be involvement of the government. How effective do you think that would be in improving AI safety? Look what do you think should happen here? Again, these companies are making these decisions themselves right now. Yeah, so I'm sympathetic to what he says he's trying to do. I don't know what his motives are. There's an aside that Sam put out a tweet last night saying we're slowing down but not stopping and kind of loosely translating what Sam was really saying. I see it as we're slowing down because we have such serious safety problems that we kind of got to where we're going to get sued out of existence. Not that he really wants to slow down. I think Sam and Dario are different on that score. I think Dario is a straighter shooter than Sam. That's my personal opinion. I think Dario actually would like to see his slow down. Sam would like to see his company IPO and they're different motivations. Having the slow down if we actually had it or a pacing might be a good thing.

22:46Having external auditors with transparent access to what's going on would be a great thing. The fact that he mentioned a company that's, I guess it's a nonprofit that's kind of in his neighborhood and in his social orbit makes me a little bit nervous. Right, me. I think a very fine organization but it should not be just them and it should not just be the companies that decide all of us so ultimately what we really need although it's hard to do in this political environment is to have a bipartisan organization that is sponsored by the government not run by the companies companies might have a voice and independent scientists should have a voice because a lot of the problem here's regulatory capture is the companies basically telling the government what laws to craft in order to keep. Other companies out I had a cartoon somebody else made actually with AI in my sub stack where Dario is on top of a ladder and he's trying to cut the ladder is before anybody else could get out and that is you know part of the worries they will pick a quote independent regulator that's actually their friends the favors them and so forth i'm not saying Dario is that dvs but you know it could kind of work out that way.

23:56So the spirit of what he suggests is good but the implementation matters and must have independence in fact what I have counseled for years including I know on this show is having something like the FDA which would not be within the companies but would be a government organization with independent auditing that basically makes companies show that the benefits that way the risks for their products this is what we do for drugs right so. So you always going to be side effects with drugs what you have to make the case that it's worth the side effects you know any kind of AI is going to be dual use they're going to be side effects are going to be negative consequences in society. You should have to show the benefits out of the risks and you should have to show that to an independent panel and that's I think the thing that we need the most what Dario is pointing to is at least a step in that direction and you know i'm glad to see that. Of course we heard from president trump on this and he does not sound interested in regulation. We could put guardrails we can do this and that but I think you have a lot of negative forces and bring it up that should be bringing it up and they bring it up things that won't happen.

25:04He is so far very opposed AI regulation but also note that he's actually flipped flopped on that so when mythos which was an ope sorry an anthropic product came out they actually band it briefly. So trump went from no regulation to banning a very specific product for a short period of time and back to no regulation so he has flipped flop back and forth I think maybe in his heart he doesn't want regulation but you know his people told him rightly we should be concerned about this and he did shut it down unfortunately what we have right now in the United States is a mess right we have. So a opaque set of guidelines that are not available to anyone I guess aside from the companies and maybe some people in the government but not even there generally I guess if nobody knows what the policy is that's that's you know invitation to corruption or at least the appearance of corruption in any case trump was you know absolutely opposed AI regulation in his latest statement and yet actually regulated anthropic from releasing a product what was it two months ago so you know there's some inconsistency there.

26:08What I argued in the economist today was that he should actually try to make a deal on China that you know trump keeps talking about wanting to win a Nobel Prize I think making a deal with China around AI I suggested it be pitched around global cooperation might actually be a way for him to win that Nobel Prize. So ruling out any regulation out of hand I think is a mistake for Trump even on Trump's terms you know on the one hand you know he misses a chance that maybe doing something truly great and truly memorable that he might be rewarded for and on the other hand he's in a box because public is incredibly opposed to data centers the American public is incredibly opposed to data centers and now you have people like Jacob Cox and going on TV saying that you know it's going to kill us all getting a lot of exposure for that and so now that now the public is even more suspicious. So if you're not aware of AI than it already was you have people like ban and Bernie Sanders and Cox and doing an event tomorrow which is like was not on my bingo card and you know that's not great for Trump either right there basically creating a new populist movement to fight his populist movement on AI he's going to lose this Trump doesn't really have an argument here Trump's argument is it's all going to be fine and you know a lot of people are not believing it's all going to be fine and even me I'm critical of Cox but I don't believe it's all going to be fine either you know I'm trying to.

27:26I'm trying to strike an intermediate position which I think is what people should be doing and so this position of it's all just going to be okay I don't think people are going to believe that and it's a huge risk right because it could be even before the midterms that there is some huge cyber act that for example takes down electrical grids that. Right. So there's a lot of casualties in hospitals let's say like you can imagine a lot of scenarios like that or not I think all that out of reasonable probability so let's say that happens before the midterms like some huge hospital chains you know loses its computers and you know hundreds maybe thousands of people die two weeks before the election like that's not going to serve Trump. Dario also wrote about China in his letter and he wrote that letting China pull ahead would create a significant national security risk how much of a concern do you think that really is like can he even strike a deal with China even if you wanted to.

28:35So one question is whether China would be willing to make a deal another is how much does this race matter and is the right way of framing this to happy race so I would argue first of all that China has made multiple contradictory noises just like the United States and my read on it is that they're willing to have a conversation whether a deal can be made is more complicated I think China in a statement that I saw this morning from the minister there is worried. For example about cybersecurity and one of the things that I emphasized in the economist piece that came out just before I saw that was making deals around cyber security so maybe we can't get a deal on pausing but maybe we can get a deal on cyber security so that we at least keep each other informed about what's happening we could also make a deal around medicine is a lot of cooperation we could do so there are multiple different deals that we could consider in some might be possible and some might not. One that we cannot possibly make is neither side is going to agree to not use autonomous weapons like that's just a waste of time to try to persuade the military is the world to do it like I would like to see a world without autonomous weapons but it's not it's not in the cards right now but there are things that might be in the cards it's even possible that a pause or at least a temporary pause might actually be something they would agree on enforce from this complicated but it's not out of the question because any rational government that is in the

30:01power wants to stay in power and AI is a threat to staying in power right so generative AI at the very least can cause mayhem to infrastructure and that'll you know piss off the citizens if you can't keep your infrastructure going right so China at least actually sees that and maybe like that's actually sees that a little bit to there is a risk to every world government from AI right now that it will undermine infrastructure in all kinds of different ways electrical infrastructure undermining hospitals undermining banking and financing I don't think it's possible to take down the whole internet the way Darry talked about but short of taking down the whole internet is certainly possible that you could for example take down a large bank that will cause a lot of problems you could maybe take down the government infrastructure so that is incentive for the countries to make a deal so that was part of your question and another thing I want to emphasize is Darry was kind of tripping over his own feet with his China hawkness because we can't really have a slow down if China doesn't also have us

31:07Well I say that we're leading China in AI we're most sophisticated country in the world and frankly I want to keep it that way because whoever wins AI wins either you make a deal or you go full steam ahead if you go full steam ahead you know I think it's a safe bet that for example infrastructure will be successfully attacked a bunch of times and so forth so it's complicated I would argue that the race so far has not done anybody except Jensen Wang a lot of good race has been wonderful for Nvidia because a lot of people have bought a lot of chips but most companies using this stuff I think are using it out of a fear of missing out and not seeing return on investment the American economy is so heavily tied to this right now that if it falls apart which it could you know there would be really could be devastating for the economy right like if open AI doesn't make its IPO and Nvidia collapses and Oracle has trouble and then the banks that finance them have trouble and so forth you can see this kind of panoply of effects that might be quite serious and so I think the race has already had consequences in terms of putting us in a precarious state and then on the other hand it was a race that was never going to be won because

32:15everybody knew the recipe from day one for how to build large language models they're all building the same technology now China has caught up open AI once had a big lead but anthropic caught up Google mostly caught up etc everybody's building the same thing that means there's very small profit margins so like do you really want to win this race I mean a better model here everybody's thinking Google because Google has a monopoly but nobody's going to have a monopoly the better model is air airlines who wants to run an airline and who wants to you know base their entire economy around an airline like airlines don't really make that much money they're terrible way to make a living and you know I mean obviously we need some and we need some large language models we shouldn't get out of the business all together but you know building our entire economy around it I think is kind of crazy and when people start to build these systems on locally like on their home computers as home computers get better and these things get more efficient and so forth I think all these data centers are going to turned out to be a mistake and so my view is that this race is not turned out that well and I'll say one other thing which is I remember the same kind of panic around GPT 5 after GPT 4 came out people said oh my god what if China gets GPT 5 first and what I said

33:28possibly on your show was well so what if they get GPT 5 first what are they going to do with it right boilerplate memos faster than us so we got GPT 5 faster than them it had no real consequence in the world it's not like you know we won the war in Iran because we got to GPT 5 first or GPT 6 you know 6 weeks ahead of China's equivalent to that so like there's not a lot positive coming out of this race as far as I can say and can see and that's why I've been trying to get people to reframe and ask maybe there's a different option the different option is maybe we work together make a global good and actually use it for some of the positive things I mean like Dario talks about curing cancer which is naive because cancer is actually very complicated and requires clinical trials and so forth but we could actually collaborate on things like hearing cancer with China which is actually getting good at cancer research so like there is this race dynamic that you know suits the company like Dario's because they want to sell a lot of stuff and so forth but there are other ways of

34:28thinking about this and we owe it to ourselves to at least try to think about those Gary this feels like a pretty good place for us to end this is a really helpful conversation thank you thank you for all your calibration it's always great to have you on I calibrated as well as I could I'm sorry that it didn't set all your fears to rest all right that is all for today I'm Jamie Poisson thanks so much for listening talk to you tomorrow for more CBC podcasts go to CBC dot C.A slash podcasts