Washington Hones in on AI Safety, What Are AI Doomers Proposing | Diet TBPN

0:00What are the AI dooms actually proposing? That's the question I was trying to answer this morning for a while. A year sentences. That's one AI 2027 predicted that this month, late in 2026, Congress would wake up and that is in the journal. Congress is suddenly waking up to the AI Doomsday threat. And so this is happening all over the place. Was it Matt Damon who was called on TMZ being investigated about his thoughts on AI risk. There's now protests. I believe AI 2027 predicts a 10,000 person anti-AI protest by the end of the year. I was trying to figure out how big this protest in Chapel Hill, North Carolina earlier this month was. I think it came in sub 10k but certainly tracking to it. It was about 150 people at this protest in Chapel Hill.

1:02But just two ooms away from the prediction from AI 2027. So I wanted to dig into what the actual proposal is because the joke for a while has been just like everyone when they're pressed on this. They just say we got to talk about it. We got to talk about this. And the actual proposal is super concrete in AI 2040. And it's very interesting just to hear about how they want to slow things down. So the thing that I think a lot of people online who are like, yay, an anti-AI sentiment that's going viral are going to be depressed about is that like this is not stop AI at all. AI 2040 is like keep the inference flowing. The current models are great. Also, let's keep doing capabilities research. But we just want to reach super intelligence by 2040 instead of 2028 where we're not necessarily prepared. So they want it to be highly controlled by governments, nation states, highly secured. And there's a whole bunch of very, very tactical recommendations that they make around that.

2:02So the first mechanism is an AI pause. They want to pause training. They don't want to do any more new frontier training runs or R&D experiments. And to enforce this, they're calling for to apply inference only verification to essentially all major AI data centers. So anyone who has more than $10,000 H100 equivalents, roughly $100 million of equipment. And that is like pretty easy to figure out. There's like big building over there. Let's send the inspector inside. Oh, says Nvidia and all these chips. Count them up. There's over $10,000 of them. You got to apply for this permit. You got to tell us what you're doing, right? Very easy to enforce at least in the United States. And with an international body, you could kind of do the same thing internationally. So the whole goal, you can only inference the current models and you got to verify your workloads with an independent auditor. Probably the government. Maybe there's some sort of non-governmental organization that's doing this. You know, there's a whole bunch of different solutions that you can pull from across nuclear and non-proliferation work that's happened in the past.

3:08Major countries, they want them to declare AI compute inventories. Tell everyone not just your local population, but also the international community. How many how many warheads you got? How many H100 equivalents do you have? Where are they? Everyone shares this. That's going to be a really tough sell because international agreements are really, really tough sales. It's much easier to have a ground swell of support for something that happens in America. America changes. We have a system that we don't really have the international rules to quickly implement that in a way that doesn't allow for a lot of defect. They want to physically remove high bandwidth East West networking inside of data centers. So you can't do large distributed training runs, but you can still do inference for any new AI R&D data centers they want. Entirely new facilities built from scratch with nation state level physical security and verification. But for the next run and the one after that and the one after that as it gets crazier, we want it in a new building built inside a fairty cage.

4:12So you can't communicate it from the outside. We want a bunch of physical controls. It's like going to a nuclear facility, highly verified who gets in the building and when air gap communications. This is one interesting proposal that they have that really shows how deep they thought this through. They want the R&D data center to be connected externally. If you want to communicate with it and you want to tell it what to do, okay, train the next running or do whatever, they will have a bandwidth capped connection at one meg per second. So you can send little instructions, but if you say send me the weights because I'm taking them somewhere else, it would take you like five years to actually trade it. So interesting like hardware solution to this. How do you actually go and implement that? There's going to be a whole bunch of other things. But interesting that they're thinking about like the width of the pipe. So it would be very obvious if you're stealing the model weights because it's like wait, this one meg pipe has been at bolt tilt for months.

5:13What's going on here? Someone's taken the stuff out of the data center. They want, when frontier model weights move from an R&D facility to an inference facility, they want it to be placed on physical storage devices encrypted independently by both the US and China. So both countries have to sign off and physically escorted by representatives of both countries to the destination. It's Tall Order. One's Tall Order for sure. And they actually want frontier models to be made deliberately larger than compute optimal. So they want the weights to be 100 terabytes instead of honing them down to something that's just one terabyte that could actually be moved around a little bit easier. Yeah. First of all, a lot of this tracks with the kind of regulation that we had pushed for beginning about two years ago, round podcast, wanting podcast studios to be air-gapped, wanting fair day cages, round podcast studios. A locked briefcase with an SMB-7, SM7B in it and in order to unlock it, Patrick or Sean Ossi and David Senna are both need to give you codes to independently verify that this podcast is worthy of being recorded.

6:19I like that one. Yeah. It just makes sense. High level value valve they see as being like the most effective in controlling the speed of capability improvements is compute caps. So there is a world and we're going to the Bernie Sanders thing because it's already getting like sort of twisted, but the big hammer is just chip controls and data center build out slow down. That's the easiest thing and I think that's like the biggest valve that they're going to be there. We're going to see twisted around to actually slow down capabilities. And so the goal is to allow models to get better mainly by adding hardware rather than inventing better algorithms, which can leak to secret projects. So the goal is like, okay, well, we know that this model is capable of this. So we want this much compute over here. Okay, you've done well. We're allocating more compute as opposed to this one weird trick that I do mersate. Yeah. So the goal here is not to go back to time because when you look at, I mean, anytime you have, you know, really, really hardcore government regulation and international coordination around issues like this, you're going to have a bunch of unintended consequences.

7:31Yep. And one thing that feels obvious around this, if these policies were to be rolled out, is that you would effectively create an incentive for millions of individuals or groups globally to be in secret, like trying to find entirely new breakthroughs that are. Yeah. Yeah. And again, this is a sense of already exists, but it kind of pushes a lot of the idea that humans are just going to be like, oh, I'm not, I'm no longer going to try to create the God model because. Yeah. Like there's this, you know, big global organization that's that's sort of policing it. Yeah. I just. I mean, that's the same thing we see with nuclear non-proliferation. There's always a discussion about what countries are getting the bomb and when and how far along are they and wars break out over this and. Yeah, like the game's not over just because you create a frame work, but ultimately GPUs and computers are.

8:34Much more wide, you know, infinitely more widespread than nuclear materials. Yeah, but you still got a Marshall them all together. Yes, there's some weird scenario where there's a Python script that's a GI that can run on your laptop. But I think most people are convinced, at least in this crowd, that scale is a prerequisite. And I mean, we were joking about like it would be so like we SSI Ilya Setskivriars knew that you know lab is recently got a big cluster from Nvidia. And we were like the most bullish thing you could do if you're this secretive neolab would be like we're actually selling our compute because we've discovered a more compute optimal way to reach a GI and we don't need a lot of compute. But of course, even Ilya is like it's time to scale up. I need more compute because it seems like even if he's taking a completely orthogonal approach to you know innovation and research, he still needs a lot of compute. And so it does it does feel like everyone is sort of with the consensus that it's going to be a big building with a lot of energy, big heat signature, definitely visible from space and pretty simple to track.

9:42At least in the short term until people start building crazy underground facilities and then you're back to you know nuclear non proliferation, but their goal is at least to like try try. Yeah, and then the other side of this is not a does it does AI development actually become something closer to the Manhattan project where you know, so a lot of people 100, 100 researchers are working with the government in secret because you can't just assume that other countries are going to. So you're going to do any of these things. Yeah, and but in in general, I think the proposal is don't go back in time. Definitely not stop everything in its tracks. It's a slow down with the goal of scaling gradually. They do actually want to reach super intelligence. They just want to do it by 2040 hence the name of the project. So the goal is. Gradually scale into top human expert capability around 2035 now a lot of people are saying, oh, we might get this by 2029 2028 2027 and they see that as too fast. So they want to push that out to 2035 then wait five years with a GI and then unlock super intelligence in 2040.

10:55This is their initial proposal. Of course, there's a lot that could change over the next decade. And and I think I think like to zoom out overall, if you're worried about X risk, the AI 2040 plan does feel like a concrete path towards slowing down the conversation definitely gets dragged down into p-dume estimates and trying to narrow down exactly how a human extinction scenario plays out. And that can be that can I feel like that's almost a side show because in a democratic society in amongst humanity, like it doesn't really matter the mechanics of getting to 10% p-dume or any of those. So if everyone feels that way, something will happen. This is a concrete plan of what that might look like and that's valuable to understand in this case. So for the safety skeptics, it's easy to see how this level of control over what you can do with computers is authoritarian or anti libertarian. Even if we're talking about 100 million dollar computers, there's a lot of people that say like I should be able to do math on my computer. I can do whatever I want. Let me do cool things. I'm excited about this. That limits your freedom. It might create regulatory capture for a few major players. It might crash the stock market or delay economic gains that come in the good ending where alignment is solved and X risk plummets. You can imagine a situation where in a few years, if X risk fades into the background, you're like, yes, there's still a risk.

12:19But it's the same risk that we face every day with an asteroid hitting Earth. It doesn't really change anyone's behavior. That would be sort of the good ending in my opinion. So it's a balancing act and for most of these slow down proposals, I personally have a hard time blackpelling about them in the sense of like, if all of this gets implemented, how frustrated will I be? Like the models are good. I would like better models. I want safe models. But at the same time, like there is this massive capability overhang, the current models can do a lot of interesting work. We're finding new uses, even for like non-leading edge models. There's a lot that can be done. So I don't believe the doom dooms who are doom about what the dooms are planning. I find that unconvincing right now. But people are starting to lay it out more. Brad Gerson, Jensen Wong, David Sachs, are talking about the other side of this equation. There's also a big question around what does the 2040 people have a point of view on robotic and physical AGI because it seems like even if you pause efforts towards RSI, well if we add billions of robots into the world that are just running on today's model, that also presents today's like, you know.

13:41I don't think they're worried about that. I don't think they're worried about that. Million robots with GPT-6 level intelligence and years of alignment work that is currently happening. That's fine. The next thing, the super intelligence, the thing that might have its own goals, I think we're talking to someone from OpenAI's robotics team, hooking up Astro to a robot, a paintbrush, and a camera we talked about earlier painting. I don't think we're at a point where that poses a risk. It's the next model. It's the model with its own volition basically. Which a lot of people still aren't seeing. They're just like, yeah, the models keep getting better, but they seem to follow your instructions sometimes too much and then you need to worry about the paper clips in our area. But it's not that they want to do their own thing necessarily. But people are going back and forth on this. Clem over at Hugging Faces, sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change, not saying it's necessarily uninteresting or wrong per se, but let's keep things in perspective and hear from the full range of expertise across the ecosystem.

14:55Nathan Lambert says, banger. What was your take? Well, my AC guy would be like, I don't really know about that, but I just want to make sure when you're hot that we can run this AC cool. Yeah, the AC guy seems to be a lot. I don't know about all that, I don't know about all that mumbo jumbo, but when it's a hot summer day, I don't want you to be worried about the heat brother. I like that. Yeah, this is kind of an unnecessary shot at AC guys. Yeah, AC guys are important. I don't know, it's funny. What did Bernie Sanders have to say? Is this real entities will sell, sell be subject to the corporate death penalty and persons shall be, shall be subject to not more than 20 years in prison if they don't pause AI development? That seems pretty easy to comply with. I don't know. I guess how do you define AI development? Is prompt engineering AI development? Then you get caught because your model sort of did a little prompt engineering in the final stage and then your guilty of this like that. Yeah, that could be a negative knock on effect. I guess it does seem aggressive.

16:11But his overall proposal is banning artificial super intelligence so no person or entity may develop or deploy super intelligent AI systems. He defines artificial super intelligence as an artificial intelligence system that exhibits or can easily be modified to exhibit capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks. It sounds like the mission statement. It sounds like the explicit goal of like 17 different companies right now. AI system or AI systems that have sufficient capabilities to plan and execute the disempowerment of humanity. Okay, that's a good one. I like that. I don't like over throwing or undermining the US government. So strongly in favor of banning that. Posing advanced AI development until a new federal AI regulatory body is up and running and then the new cabinet level federal agency will monitor frontier AI systems at all stages of the life cycle. Supervised the removal of dangerous capabilities and supervise the destruction of artificial super intelligence were coming for it, which is similar to the corporate death penalty is a line that you don't hear a lot right usually people usually these companies just you know.

17:24Go bankrupt and wind down but corporate death. It goes pretty hard metal it's kind of it's kind of metal yeah you kind of got me with that one. Yeah rough rough rough situation we'll see where it goes. Jamie Cox over at over at fluid stack the co founder fluid stack a computer provider shared his convictions sort of pushing back on a lot of this saying. That he thinks America should build more their pro freedom pro democracy. They believe AI will bolster human flourishing. We support simple clear enforceable regulation frameworks that set simple requirements proportion to keep abilities and risk with clear responsibilities and no unnecessary barriers to competition. Yeah, the real you're going to see a lot of pushback from people who are like the like the regulatory stuff is going to be a lot of work. The regulatory stuff is going to be like these 10 companies and I'm going to be number 11 and I'm basically getting the corporate death penalty then because I didn't make the cut to be one of the regulated one of the approved companies.

18:34I'm still early in my stage so there's a lot of nervousness I'm sure but will the night is said we believe AI will make everyone rich healthy and free is novel and interesting comms from the frontier. He's he's endorsing this and I agree I like I like this I like these convictions. I think it's generally like a positive direction to move in not a direct response to the proposals that are going out but we're going to get a whole lot more of them. Where do you want to go to next over on tiktok they're sharing photo of the whistleblower and saying in every worldwide disaster movie there's a dude that looks just like this that nobody listen to. He really does look like he does look like an actor here yeah he looks good but people are people are all and please stop calling him scary potter I've been seeing people as a single level of run acts calling him scary potter it's the goal goals to is to wake up China wake up Congress wake up everyone.

19:37Door dash is entered the conversation. They say two years at door dash how do not say this lightly we are extremely close to the breeder arriving before you decide you want it we are not we are not asking for a band we're asking for a pause I don't know why they would ask for a pause. Yeah that seems like a very very aligned to humanity yeah into their business yeah which I think is fantastic the doom is very much contained to the frontier lab work everyone in the application layer who's applying the models diffusing them they're like I just I can't get this thing to work right I got to get I got to get forward deployed engineers to teach people how to use this thing everyone deeper and actually Jim Jim rainbow over Jim or Riley sure Jim Riley yeah. Jim. Oh Riley out of part. Jim Riley over Charles and I I says it's a whole lot of mumbo jumbo that was your words he just he's just happy to get you know eight hours back yeah yeah yeah and then yeah everyone deeper in the supply chain like

20:40Jensen and all the different semiconductor manufacturers are are not particularly on this side and then you also have Wall Street who's just like what's the enterprise acceleration some lots of different groups around the table that need to be brought on board to this movement the discourse truly is fascinating so what is Tyler Cowan calling for he's banging the table saying bet on this Tyler cause this are you named after Tyler Cowan is that your name sake Tyler Cowan says if you have very pessimistic fears or predictions about AI name the market prices that will support or confirm them this is what taking this seriously means and I love Tyler Cowan I'm not sure this matters because if you if you you're not going to be around to collect it right yeah this is what deep deep dish yeah so Tyler why would short term existential risk affect market prices in any meaningful way spell out the exact mechanism contracts that pay out if everyone dies aren't worth anything to me

21:45Yeah I don't know yeah I think he's just he's calling for like oh you want to like make a falsifiable claim like is this am I able to tell if your claim is like true or false and so it's like very hard with these scenarios where like isn't unfalseable claim though just by definition and like you just have to like accept that move on yeah but then it's like so hard to have any like real discussion was what was nuclear any any different like the threat of nuclear apocalypse the threat of world war three this was a very motivating factor for decades most of the 20th century people made real decisions based on it based on where to do business and where where where the where the conflicts were going to be and the motivation for nuclear treaties and non-proliferation and how we you can make financial decisions like think about nuclear war right you have a you have a bunker that's like that decision you make funky or is anyone making like AI bunkers know because they think it's going to be so totalizing that the bunker actually doesn't do anything exactly yeah so if you think it's so totalizing then you don't make the bunker and so saying hey you don't have a bunker is not

22:53is not proof that the person doesn't believe what they're saying there there is the other side of this which is Paul Christiano who is has been worried about risk he recently just joined the board of open AI and and Tyler Cowan you know has this quote if you're a doomer why aren't you short the market and Paul Christiano is 2x levered long and he short the isn't he short the the bond market or US treasuries right so he is 5% of his net worth in Tesla 90% of his net worth in AI bats and 100% of his net worth in normal investments no Tesla options that sounds like a scary place with lottery ticket biases and the crazy Tesla investors and then you a laser you to cask he says am I correctly understanding your 2x levered and Paul Christiano says yeah and so he says he's personally short the US the US 30 year debt I think that just means you have a mortgage this doesn't seem like a doom based bet this seems like this bet also pays out in

23:54just like the good ending and like AI is real and delivers value so your AI bets perform well the market performs well and money slides from US debt to data centers and AI build out debt or something like that so he is putting his money where's mouth is but it doesn't feel like a representation of like doom by any means people have been creating agents of fruit flies have you seen this yes my understanding is is Google basically mapped out all of like the neurons in the fruit fruit yeah like the entire 3d they got the entire structure and now people have been able to recreate that in software in a simulation yes I think in theory you can like replicate all of the the flies like decisions or whatever it's like movements yes and this is notable because many people have said I have the mind of a fruit fly this is the end they've been they've said I have the intellect of a yeah so now they're going to put it to the test to see which performs better the simulated fruit fly or just Jordi Hayes

24:57so so now that this is out in the code is out and you can run this fruit fly in simulation however you want people are having they they're doing all sorts of experimentation so Kevin said he trapped the fruit fly in his rabbit R1 when he shakes it he can see its brain brains escape circuit light up yeah I think what what ends up being unsettling and weird about this is is if if it just just purely you being human and doing this is is probably not good for your own soul and you know imagine imagine you have a fruit fly in a box shaking it and you're like look it wants to escape yeah totally I'm not a huge fan of small insects I don't really want them around that much but when they're in my house I tried to you know if a spiders in my house even if I know it might want to take a nice bite of me yeah I'm still going to try to transport it out of my house

26:01and and put it back into the world I think it's I think is it's not good for your for your soul to be you know a merchant of death sure and yet and yet if you're playing a real-time strategy game and you highlight a bunch of soldiers in this simulation and you send them on a charge that will result in their virtual death you might not feel those soldiers opted into writing and trying okay okay okay what about it's just sitting there John so they made the fruit fly play doom what about if you play doom you are killing a demon who simulated is that immoral like well but the rule is that demon trying to kill you a lot of it comes down to like the fact that it's simulating a the actual representation of the fly makes it a lot more concrete than just oh yeah it's a it's a 3d model and a python spirit that just says if you see a if you see a character shoot at them in the simulation

27:03but we're we're clearly starting to grapple with like these odd moral questions of way if you're simulating something then the next step it's a couple of order magnitude but you get there and you can simulate a human and you can talk to that human and it would do everything the human does does that human have rights and agency is an ethical moral agent or is it merely just a simulated just a really good computer simulation it's just existing on transistors so you don't need to feel any moral weight about anything that you do to it I agree with the just the bibles based analysis that like torturing a real fly torturing a virtual fly probably just don't be in the business torture and anything you don't need to overthink it but people are the real debate here is like the question of you know RLLM's sentient are they moral is there a moral weight to the to to to synthetic intelligence to artificial intelligence that's what people are debating here and I'm sure the debate will continue who knows if it will ever be ended but they did teach it how to parallel park which I think is cool

28:09interestingly last night I had astero use computer use to play a video game that I very much enjoy playing called blotro it's sort of like a modified poker game and I don't feel like I was torturing the all I'm by that I feel like I was giving it a treat instead of doing my taxes you could just chill and play a video game it did very well it won so was not able to win and it was really fun because I would like pop in while it was using the computer and set cards kind of arm share quarterback and be like is it making the right decision right now but like a felt like a coach coaching like a kid on the center pitchers I had a funny moment last night I was on my racing simulator comparing asking J.B.T. for for to compare my time yeah to just other other you know what what would best in class be like what's beginner like et cetera and it said if you want I can help you cut cut some seconds off of this time

29:12yeah and it was like why don't why don't you take a video of a full lap so that and I'll analyze before you and I was like yeah I do I bet I bet you like just watch watch you know watch track and watch track footage for the soon it's gonna be like you want to just get in the seat yeah I just take over because like I do yeah no computer use in I racing is something I'm gonna experiment with this I'm down I'm down take take half my quota my monthly quota just play games chill do whatever it's nice treat I've got some bank resets yeah it'll put to work wonderful week wonderful week short week yeah but Monday I'm do you say favor and go ahead and have the best weekend best we're entire life the best weekend of your life do it do it put the pieces together make it happen we'll see you on Monday leave us five stars and have a podcast and Spotify sign up for a newsletter tvpm.com