Databricks CEO on AI Pacing, Cyber Risk, and the Enterprise

0:00As a business leader, there's a tragedy of the comments. If you want to stop, if you want to go slower, why don't you go slower? Like, I'm competing, I want to win. There's almost two camps. There's one camp which believes that this actually is an engineering problem. And there's others which actually believe you have to slow it down. Humans don't respond fast enough to the attacks that are happening. You need to automate all of those. And also organizations are actually not supposed to do that. It's RSI and recursive self-improvement that the labs are doing, leading us there. That's the big question. Something Elon said, this is some elaborate for D-Chess because on the one hand you're saying all of humanity will die. On the other hand, you're saying, hey, what do you want for your IPO allocation? For the first time ever, a company at scale last week said that they're moving from the frontier models to GLM. Do you think that that's a trend or anything? That's just like a one-off anecdote. AI may already be smart enough for the enterprise. The problem is that it doesn't understand your company. Databricks CEO Ali Godsee joins A16Z's Martín Casado and Sarah Wang

1:02to discuss what's actually holding back AI adoption. And why the answer may have less to do with building smarter models and more to do with giving them the right context. They also debate the push to pace frontier AI, recursive self-improvement, and where the risks are real today. Ali argues that superintelligence remains far from what we currently see. While cybersecurity is an immediate problem as agents make attacks faster and harder for human security teams to keep up with. And they get into what Databricks has learned using AI internally from building an organizational ontology to managing token costs and choosing different models for different jobs. Thank you for being here, Ali. Super excited. So we obviously want to get to Databricks, but there is a broader conversation going on right now about AI. And Dario's waiting, Yacht Cubs waiting, Elas waiting. But we want to hear what Ali Godsee thinks. In terms of, you know, if you called the topic, broadly speaking, pacing the frontier, etc.

2:03What is your strongest agreement with what's being out there? Where do you disagree? And maybe where is their nuance that's not being captured? Yeah, happy to cover it. And me and Martin argue a lot. So I'm sure that's not going to take long. I'll try to relate it in this. Yeah, try to stay calm. But why I do think first and foremost that there may be a agree on this, that leaders have responsibility to not freak people out unnecessarily, unless there's really, really good reason. And I think, you know, there's always different people in society that are different places, you know, in their mind space. So, you know, talking about these kind of existential risks and, you know, scenarios where all of humanity is going to be wiped out. I think it's irresponsible. Like it can tip a lot of people over and it can cause a lot of mental health issues, unless you have something that's going to wipe people out. Yeah, as I said, yeah, if there is a actual reason for it, then, you know, that's a different story. But I think that right now, the existential risk is close to zero.

3:06So, why freak everybody out? It's not actually needed. There are risks. We'll get into it. That's probably where we disagree. But first, I think that leaders should not freak everyone out. And I mean, you know, if there's like technical nuances in how we're doing AI research and so on, well, researchers can discuss that. You don't need to every time go on TV and or blast on Twitter to millions of people that, hey, you know, I think there's like this percentage, 10% risk that all of humanity is going to be wiped out. I don't think that's like helpful for a lot of people. Actually, I think it causes a lot of harm for a lot of folks who get stressed out. And actually, I'm not in the nuances of all of this stuff and what it means. So that, I don't think we should do. I don't think it's fruitful. It doesn't really help anyone. I mean, I think this is very, very true for the general public. Yeah. Like my sister, who's great. She was a school teacher and rural areas on a on Sunday, Texan means. It said, it said, Martin, should I prepare the cabin for, you know, she's kind of a prepper anyways, but should I prepare the cabin for the AI apocalypse? You know, I've got water set up.

4:07Like when are you showing up? I'm like, hold on. Yeah. So clearly, this is kind of spilled over the populace, which I agree is unnecessary and has blowback. I think there's a second one, which is, I don't know if you saw like walking in here. I was checking X and Elizabeth Warren just talked about pausing all of AI development. That of course is on the co-tails of Bernie, who's also working with, Benin, like Steve Benin, like a job. So now, so in addition to like, you know, just scaring people, the federal complex is now spinning up. And I think that could be actually quite contrary to the actual goals of the message. And so there's more than just, you know, I think public hysteria at stake here. Yeah, there's a lot of politics going on, but I'm like in all of these groups and, you know, I see both sides. There's heavy politics happening on both sides, we should say. Like, right? Oh, this is happening on both sides. This is, no, this is a, this is a, I think both parties that do not include Trump himself.

5:09Agreed that, that AI should be constrained at some level. I'm talking about the other side of this argument as well. Let me give you an example, okay? Even Greg Abbott, right? Like even Greg Abbott was like, you know, you can't have data centers in Texas. Well, I'm not talking about politicians. I'm talking about there's politics going on on both sides, right? Right. There is politics on the business side. People who want to see great IPOs and they want to get returns on their investments. And they're like, don't mess up my IPO. And they want to get like, hey, can everybody just shut up so that we can get our money back? So there's that. And they're, you know, they have resources and they're using them. And, you know, so there's politics on that side. And those are like, not, they're not sitting quietly and not doing anything. And they can pull strings and they have connections. On the other side, there's a lot of people that are like, okay, how do we weaponize this? This is awesome. Yeah, yeah, yeah. Let's, you know, let's weaponize this one. Let's plant this. If I, you know, let's pump these, you know, threads. Let's, let's talk to the specific leverage point that everybody is using. Because I actually think that this is like a classic case of a PR messed up. And it's not just the doom and gloomy type stuff. So here's the PR messed up, I think, which is, like, it is not unusual for industries to try and regulate themselves.

6:14It's just not, right? And I think seeing like security and safety is important. It is with every techie puck and we want to have some oversight. That was very, very sensible. The problem is, is just couching this notion of pacing. And there's, there's a number of issues with pacing. First off, it's orthogonal to safety and security. Like you can slowly build a weapon. Like it was not different than building a weapon. People don't feel it's, it's, it's genuine because like these companies have been at a dead run. If you're still buying more compute to be even better. No, no, I mean, no, I mean, like, they just haven't done it. They haven't done it historically. But also, like, kind of, kind of feels like this kind of almost milk toast capitulation to the pos people. So you're like, well, you say pos, well, I say pacing, which is almost like pos, but it's not like pos. So like they, they chose, they chose this kind of like flag to follow around pacing. But if you actually read, did you read the, the, the document, the direct, it's a totally sensible dog. May I write it? Yeah. It just says nothing to do with pacing, right? And so I, I honestly, no, I just mentioned it. Look, see, I, I kind of a little bit disagree.

7:15I look, there's a tragedy of the comments. There's this like, hey, if you want to stop, if you want to go slower, why don't you go slower? Why are you write articles? There's a lot of people making that argument. But no, I mean, as a business leader, I understand that there's a tragedy of the comments. Like I'm competing. I want to win, you know, and you're trying to have, I'm going to, there's also the market equilibrium, which suggests that pacing is probably in practical anyways. Yeah. So I'm just saying that, you know, so it makes kind of sense for people to say, hey, if you guys don't stop this, this tragedy of the comments are going to continue. I'm not going to stop racing because, you know, there is IPOs at stake. There is a competition at stake. There's also some animosity between the people. So like, I'm not going to stop unilaterally. I'll be a sucker, you know, why don't you stop first? So then they're saying, hey, can you come in and stop us? But, you know, I think that you could also make the argument that if you look at the hugging face, opening an incident that, but I think these companies are great. And I think they are probably investing a lot of resources. But it's very clear from, if you read what happened, is that there weren't monitoring every token coming out and having it, you know, there were just like running these RL experiments.

8:16And then after the fact coming in and checking up, it will happen. So they should have paced. They should have been much lower in that particular incident, right? I just want to just quibble on syntax, but like words matter with PR, right? So let's take the hugging face incident. When I read that, you know, my reaction was, was not, oh, opening an accident pace. Like, dude, fucking secure your thing, right? Like do security controls like we always have done. It is pacing though. It is pacing. It is pacing. Like it is in the history of the internet, we had all of these things where we're like, let's pace the growth. The inner way. Let's do security, let's do control. Let's do like what I mean. You should do that. But it is pacing in a sense that look, I face this all the time. I have a legal department at the inter-rix, I have a security department at the inter-rix. And you know, they are always like, okay, slow everything down for everything. Not AI. Like literally every little thing. Like, oh, you're going to go on a podcast. Well, what's the script for it? What are you going to say? And let's review that. And you know, what's illegal? You cannot say this. You can say that. Everything you say have to do materially true. Are these pacing people in the room with us? Yeah, that's right. So, you know, so you're running our experiment.

9:18You're training the next model. Should the security team be there and look at like run all their monitors and look at every day? I mean, like millions of hours of GPU hours of tokens were produced. And these agents were running, you know, a mock in the sandboxes. It would have slowed the numbs significantly. If you had security teams sit there and look at all of the stuff. I agree. Now, and they're saying, hey, like, you know, if we do that, it will slow us down. And I'm not sure the other side is doing that. So can you guys come in and slow us down? Like, just tell us. Like put some guardrains around us. We'll happily then follow the rules and do the secure thing. Otherwise, it doesn't make sense because we will get our. I just think like nuanced second order words don't work when like people are really afraid. You're like, I'm going to pace. And therefore things like these don't happen. I literally think we should just been like safety security is paramount. We're going to put in these controls. Like, that's the important thing. And I do think that nuance actually got lost. If you look at what Zach said, do you agree with that? I thought that was like, we're going to pace ourselves. We're going to put in security. Like the reason we released this later is because of security. Well, I thought was so great about the. Sock is like he was very focused on like security, safety and self regulation.

10:23Dial, dials first five words or whatever. Like, we need to pace the frontier, right? It just put you in a very different mindset than what he could have said is we need to secure the frontier. Fine, we need safety. I mean, they're like at some level, I think they're trying to optimize both for the dooms, which cost for pause and for politicians. And they kind of didn't satisfy either. Because they do. But those people are actually freaking out inside the labs. And there are a lot of safety people that are freaking out generally. By the way, and not all of them are you people and so on. And the people are like, hey, they're surprised, right? Right. But here's the thing is that using the worst pace doesn't help either of them. I think it's like literally, you're like, you're like, you're trying to find this. Like he, he, he, he, pace is like the uncanny valley of making the doomer people unhappy and the policy people are happy. Because the doomer people like, that's not a pause. This is pacing. Yeah. And, you know, everybody else is like, well, like this is, you know, this isn't really you're not going to do it anyways. And you're not supposed to be security. So again, independent of what we should do, which we should talk about. I just think that the way it was presented was just bad.

11:24And it just didn't work. And that's what I'm going to blow back at. These guys are, you know, they're not trained, you know, PR people. You know, and yes, some of this stuff. I agree with you. I mean, I agree with the, I mean, I agree with the, I agree with the core premise that we shouldn't freak the public out. I think that it's a central risk right now is close to zero. You know, but let's talk about the core thing, which is the, the fact that, you know, anyone who's doing big reinforcement learning runs. And they're giving it a reward function. And they reward functions or unleashing, you know, saying, hey, here's like a, you know, 10,000 agents. And here's $100 million. Let's put them in parallel and let them run on a gigantic cluster for a month or two. Try to solve anything and it doesn't need to be a security thing. It could be like, do anything, you know, solve this map puzzle. Really bad things can happen. Really bad things, meaning things get hacked. And it has, you know, cyber is the primary one. Right? That is real, right? I think of this making it, hey, this is an existential risk and so on, which I think was a mistake. I think it's not good to scare the public that way.

12:24I think it's become something that everyone, not just your sister, everybody around the planet is like now talking about. I've had all kinds of people that never care about this stuff. And they find this extremely boring. Ping me and say, what do you really actually think about this? This is really important in my counts. Now I'm starting to worry about it. So then it becomes a political issue and we have elections here coming up. But there's elections all around the world. So you're going to see, they're not going to sit still in other parts of the world either. But I think that's our responsibility to talk about this in a balanced way and actually expose the risks. I think that's super intelligence, that idea from that book is very, very far away. I don't see any evidence that we're actually marching towards that or that's going to happen. Apparently some people, yeah, apparently some people with labs are freaked out that maybe there's progress towards that. And I think it comes from RSI recursive self improvement, the models improving themselves. I would love to understand how much, what have they seen something we don't know? There's four criteria. If those four things are happening, I would love to understand them. One is, if we end up in a situation where following four conditions are happening,

13:26which is the next model requires less resources, less GPUs to train. And super linearly, not just like tiny little bit. The next model takes less time to train as well. So it's a second condition. Third, accuracy of the model, the intelligence is increasing. And fourth, we can do the former three again and again and again. It's not just us. All those at the same time, right? All at the same time, not just any of them. Yeah, all four are happening. Then you can imagine in a way where you can, you know, because any of them that's not happening like for instance, if resources is constant, then that's okay. Because we're going to run out of hardware. So then it will pace itself. We will not have enough hardware to do that, not enough GPUs. Time the same. So it needs to be that you end up in this situation. So if you just mean that the software is writing itself, we're already there today. 90, some percent of the software in Databricks is written by AI. Does it matter if the last few percent is also written by AI? No, it doesn't matter, really that much. But if you're getting these four conditions, then you might get a speed up where the next model

14:29let's say takes half a month of time and half the resources. And it is more intelligent. And you keep doing that, you know, then you might end up in a situation where I don't know. But I don't even know if that necessarily leads you to Supreme College. That's a great conversation. But it could, so then that would be more risky. So it would be nice if they can share all that data and we can share some light and transparency on that. I actually think it's a great breakdown that you have. I don't think anyone is using that as a definition. Actually, you're right. I think there is a little bit of people freaking out about like, oh my god, the emergent behavior. Now it's creating itself and so on. But I think like, as I said, a lot of people there definition is just, hey, if I'm not even coding anymore and it's coding itself. Right. But I think they're conflating, hey, what's my value and is it scary for me versus, hey, that then means we'll get that super intelligence that 2014 theoretically was hypothesized by what's from. Well, you're putting in compute. Those are really good one that's missed in a, I think a lot of arguments on RSI, right? Because as far as we can tell, the minimum threshold for compute needed to train a good model just keeps going up.

15:30Like it was a hundred million billion. Now it's probably like five billion. And so that's a model now to train like a frontier model, right? Five to 10 billion. Right, right. Exactly. Billion billion billion. Billion billion. Versus the first time you've got a very expensive. Yeah, French is very expensive. To replicate the frontier six months later is about 120 of the cost. No, I think Sarah is a great point, which is, this is a good argument against this whole thing, which is that the next model. First of all, there's only one or two such runs a year that each of these labs do. And they take the opposite of the four criteria that I mentioned, right? Which is, it's going to take more resources, more humans involved. And it's even more brittle. And they have to build out the data centers. I mean, like the labs are not necessarily doing that. But others have to build the data centers and they have to be gigantic. And I have to get the GPUs and they have to get the networking right. They have to do the engineering to make sure that they can tolerate. Because you know, every order of magnitude, more GPUs are crammed there. You have to not worry about errors that before you didn't have to worry about. So you have to increase robustness of the, so it's like a very brittle process. And if it fails, you've squandered so much money. So they're like very, very careful with that run.

16:31And there will be multiple runs that have been botched. So it's the opposite of that that you hate. The next model is faster, cheaper, smarter, and recursive improvement. It's the opposite. It's like it's taking longer and it's more brittle and it's more people and it's harder to pull off. So I do think that is true. With respect to our side, with respect to actually cyber risks and things get hacked. We need to take it seriously. Yeah. So I'm going to have another like like, like, Miss Testier's like, I actually love your fork materials. Like literally just waiting to argue with it, but I actually think it's better. Like it is, it's better. So let me give you like, like a black box, what, like when you're dealing with these dynamic adaptive systems, like, what are you going to believe? Are you going to believe like the numbers are your lying eyes, right? So I think you kind of have to go to the numbers on these ones. So like, what are the numbers to look at? I really think you should just basically, and maybe going public is the right way to do it. Like, like if these companies continue to grow, reduce the number of people and the number of money that goes into them, then I would say something is definitely happening here.

17:32Like I do think that like you can actually black box this and take a look. But none of those indicate like they're hiring like crazy. But that's not fair. That's not fair because you know, companies are not necessarily efficient, right? So like, what if you have, I mean, opening it itself was doing like a million different activities. It's a very small team of like 10 people were doing LLM's and the LLM stuff was useful. Twitter was a lot of people. Now it's much less people. I agree. It's just another LLM test. We have to have two LLM tests. You're a LLM test, which I think is great. But then you would actually have to have a way to instrument it. Yeah. And then we should have the black box LLM test. Like, I mean, listen, if you're on Thrapik in two weeks is, you know, 12 people and they continue to grow. And they're putting out models that are increasing. Right. I think we should probably take notice of that. That's sufficient. Right. But it's not necessary condition, right? But I'm just saying that, you know, there could be that, you know, it's really the right way to do this then to look at, okay, the pre-training and the post-training that's been doing that's really necessary. Because they have so much resources that they might be doing a lot of other stuff they don't need to do. But they're doing it just and it can just hire people because they have infinite money and infinite. So really, the people that are training the next model is that team tiny, tiny, and it's actually getting reduced and they're doing less and less work and just AI is doing it.

18:34And the post-training and then they're all just using less GPUs. That's not the case. We've had this argument many times as an industry before. I remember when like, like, we learned how to really cluster computers. Because the mainframe was actually kind of limited by things like memory coherence. Remember that? Like, you only make it so big. And then we kind of went to the client survey area and then we didn't have the problem. And then we started creating super computers, which were like basic, just like, you know, cluster computers. And that's not what the internet happened. And that was kind of also roughly like when GPU started getting good. And do you remember that we would actually like, expert control play stations? Because we were worried that Saddam Hussein would use them to do simulation. And the arguments were very similar, which is like, these things are getting infinitely powerful. We were using them to simulate nuclear weapons, which we were, like I was. Like we can't, you know, this stuff has existential risk. Actually, they didn't use those words, but like this has the potential for like nuclear weapons or whatever. And we should stop it. And like, none of that came to path. So I think a very reasonable discussion is this time different.

19:34Yes, I know. I don't have an answer to that. But I'm VC, but you know, you're a. Yeah, I mean, look, I think I was, I'm not old enough to remember. So ignorance is bliss. Wait, wait, wait. So I can take this. I don't know. I don't recall PlayStation being legal and Saddam Hussein being. I just don't know. Maybe I was just saying. Maybe I'm just ignorant. This is like 19. Maybe I'm old Annie. Maybe it's just amnesia from age. But what ever it is, it doesn't care about the expert controls in the oil. Yeah, you know, whatever it is, I think it's it's at the different scale now, right, with AI and you know, with the, you know, what we're doing. The pace of development is so on. They are freaking out different here. I do think cyber is actually one of the biggest ones that we're going to see, right? Because it's just so much infrastructure on the planet. But way, way more than it was, whenever whatever Saddam or Xbox or whatever it was, you're talking about. I mean, like we've just interconnected way more things and they're dependent and like the planet just looks different today. From internet tech dependency interconnection.

20:38Then, you know, 30 years ago. So I just want to make this point. There's so much infrastructure as insecure, right? And if you're going to unleash these agents, they're going to find loop holes. They're going to find exploits. They're going to break in here and there. So, so this is a real risk. And you can't just, and by the way, this time it didn't do that. But you could imagine a scenario also where it starts hopping. Like it takes resources and it starts executing itself elsewhere. So it's kind of spreads like a virus a little bit. So that's a real risk. So, so, and this is pure curiosity. I promise I'm not, you know, trying to be a foil here. But like, why do you think we just haven't seen very much then? Like again, again, I'm much older than you. I remember very well. So when the, like literally when the internet, when the, when the internet came out, by this point, we had literally taken out 10% we've disabled hospitals. We've taken out critical infrastructure. We'd cause tens of billions of dollars in economic damages from worms. Like all of that had already happened. And to your point, we had much less build out.

21:38You know, less of the, the economy was on it. And so, you know, AI has so many people that want to find risks and threats. We're running so fast, so much money has been poured into it. And we haven't seen anything commensurate with the early days of worms. What does that disconnect? Yeah. Look, so I do remember that those days. But it's at the same time. Yeah. So look, I would just say that I am slipping well at night. And I don't think there's existential risk right now. I do think there's a lot of infrastructure needs to be secured. Yeah. We have a product in the market, in the detection market, Lake Watch, that helps you do detections. And the space is just there is moving so fast. Because, you know, you just have these sock teams, security operations center people that would, you know, look at what the intrusions are happening, how are we being attacked and so on. And now the humans just can't keep up. So this is the whole space, security cyber space is being transitioned into fully automated using agents for detection on the other side.

22:41If we don't do that, I mean, now we're rushing, we are rushing the industry is rushing to do that super, super fast. If we don't do that, I do think you will start seeing those kind of things like sites going down, you know, you know, whole systems that stop working for a while. And there will be consequences, not existential, but economic damage and, you know, people getting hurt and so on could happen. So we just have to raise very, very fast to do all of those things. It's just you don't have the humans don't respond fast enough to the attacks that are happening. So you just, you need to automate all of those. And also organizations are actually not as close to doing that. The banks are doing it. Some of the people that are super security conscious are doing it. But most of the industry today is running with old school security operation centers and people that are waking up every day. And there's like hundreds of emails of detections that have fired. Many of them are just false positives. So you don't need to ignore them. But some of them are not. They just don't have time to go through those. And you need to identify that. You need to have threat hunting. That's automated where you're actually attacking your own systems automatically with agents and so on.

23:41It hasn't happened. So I do think like if we just say, hey, this is just like the internet in the early days, you know, bad things are going to happen. So there is a race going on. It was actually very surprised. Earlier this morning I was on a conference like I feel like you and I are like pretty close to be prognostic like I feel like I know a fair bit about Databricks. I was on a call this morning where a founder was basically like, yeah, listen, like, you know, we're doing all of this like observability agent threat detection. I'm using data, but I didn't even know that you you had this offering like quite frankly. So like, I mean, this is just from an education staff. I'm like, how extensive have you gotten in like the agent AI observability security safety thing? Yeah, I mean, we haven't talked this year at RSA actually with Ben Horvitz. But the issue is that data and AI is blending with cyber. These two markets are collapsing because I think you know, yeah. And the reason they're collapsing is that it used to be like, okay, we have like data and AI, the kind of stuff Databricks and these kind of companies used to do, which is like, okay, we have a bunch of data and you run AI and machine learning and that's a slip separately.

24:45And then you have the cyber world. Cyber world is, you know, we wanted to take something bad. It's like bad people are trying to hack us if bad people are doing things, we need to detect that. Okay. But now on the data and AI side, we have agents running internally in the company. People are having agents running and the agents are also like doing things with other people's agents and they're producing a lot of data. Logs, trails, you know, fingerprints that are being left. And so, you know, then you have internally these agents that are doing that. So these worlds start merging more and more, which is like, okay, well, all the data that's being produced needs to be analyzed. And the scale at which you need to do that is just like many, many orders of magnitude more than just one or two years ago. So things have changed dramatically. Like 2018, 19, the time it would take from, you know, a CVE vulnerability being sort of, published until you see it actually be weaponized in the industry would be like two, three years. That went down to, you know, 2022 significantly, but it was still like eight, nine months. So that's kind of fun.

25:45You have eight, nine months from a vulnerability to that was 2022. Now, if you look at this curve from 2022 until now, now it's down to like basically hours. So it's like down to like basically no time. Like things get immediately weaponized. So you need to just do it in it automated with the data and AI sort of platform approach. So these markets I'm going to argue are just going to collapse actually. So the very specific question, I actually think a lot of like it to pull back on the existential, ex-risk discussion. It feels like there's almost two camps. Yes, there's one camp which believes that this actually is an engineering problem. And like companies like, yeah, like Databricks can solve it. And they can solve it through product and through engineering solutions and through services. And so like we just as an industry can solve that problem. Yeah. And there's others which actually believe seems to me that there is no engineering solution. You have to slow it down. You know, you have to use regulation. It's more like a nuclear weapon, et cetera. So like does this mean you believe it as an engineering problem? Or are you not quite comfortable saying that?

26:46Yeah, because why it was testers? Why why don't you even buy Databricks? Man, let's just put this stuff in the national app. Just pay the only solution. I'm just going to just pay for themselves a little bit. They don't find the bad guys. There is no line of inquiry ever that gets to pacing. I don't think I think it's like you pause it or like you like to solve it. What we've learned here is that Martin really hits the word facing. I don't know if you use that word with you ever again. Okay. Clearly, do you really know it? More but first straight in mind. Yeah. Is it an engineering problem that can be solved by engineers or is there more to it? I actually think which problem are we talking about? There's two separate problems that I think are being completed. There is the super intelligence problem. Yeah. And I think a lot of this comes from both Strem 2014, Super Intelligence book. And if you look at the definitions, I think people don't have this clear definitions of what Super Intelligence, if you read his book, those definitions are kind of crazy. So I think what he had in mind when he said Super Intelligence is, you know, AIs that, I don't know, I don't know what the examples were. Yeah, something like the right to whole PhD thesis with novel like peer reviewed stuff in a couple seconds.

27:48And they can do like millennia work the thought, you know, in like instantaneously. And, you know, so this is like the level of, you know, how fast they are, how intelligent they are. So it's like I can learn. Yeah. Yeah. Yeah. I mean, yeah. But it's just many, many, many orders of magnitude. Right. It's like it's just the scale of the problems is completely different. So if such a thing exists, do I think it's just an engineering problem to solve? No, I think that's actually a very, if such a thing would happen, that would be very, that would be very substantial. Of course. And that's what I really agree on. So I think that's being mixed with, now we have agents that are nowhere near that. It's not even like there's nothing like that. And we don't have anything towards that path right now. But these agents are capable. And you can do something with them that you could never do before in a history of mankind. So I do think an efflection point has happened. Something has changed, which is we could get, we have good security researchers at Databricks. But I could never say let's get 10,000 of them in a sandbox for a month and have them do $100 million

28:49of salary wage work. We can do that now. We just turn on a button and we can get 100,000 of them or mathematics. Like we can say, hey, you know, we want to solve a fee like a conjecture. Okay, let's get pretty good mathematicians, but let's have 10,000 of them collaborate. You know, and then you can like make very fast progress. So this I think is an engine. This leads to all these cyber risk. I think cyber is the major problem here. This is I think you can solve with engineering. And I think it's like we are working on it. Many others are working on it. There's still risks. They're not existential. I think we should do it. There is the super intelligence thing. That's the thing that could do right a novel PhD thesis or like reason intuitively and 11 dimensional space physics instantaneously without writing anything down. Something humans can't do. Like that kind of super intelligence. The question is, is RSI and recursive self improvement that the labs are doing? Leading us there. Are we going to get there trying to do trying to do? Is that what's going to happen? Yeah. And how fast is that going to happen? That's the big question. And they've suggested that, you know, hey, we should have inspectors that come in and look at what we're doing.

29:52And there's a good idea. Have them go in there and get the data. I would love to like the question is who are the inspectors? Because you can like you can stack right? You can stack that. Who do you think you're doing? Yeah, there's a bunch of people that like on either camp actually, I wouldn't tear it. If they're the inspectors, I would not be very impressed by what they say. Because they've already made up their minds even before they are. They would go in there. Right. Exactly. But let's say like as an example, if John Bakun, who was one of the inventors of this, you know, deep neural-electable technology, right? Wanted pioneers. If he said, hey, there's nothing to see here. There's no risk. You know, I'm paraphrasing him. This is nothing. This is super intelligent. This is just nonsense. Keep on going. Go fast, fast, fast. None of us would believe it. I'm putting words in the bathroom. Yeah, I'm not exactly aware of it. Now, if he was one of the inspectors and he went in there and he had the look and he came out and he said, hey, I've looked and it's just what I said. There's nothing to see here. Just keep going. I would feel very good about that. That would take a while. I would feel very, or if he comes out and says, oh my God, you know, I, you know, he's wobbling and he would change his mind a little bit.

30:52That would also have a lot of interesting signals. So I think it comes down to who pick as inspectors and it's a good idea. Let's have some of them and pick a diverse set of people so that we can get different nuanced points of view. What do you think about this kind of Elon Musk view, which is like, it's less third party. It's more, it feels like there's kind of three proposals. Like the opening eye and the topic one is a third party. Yeah. The Elon Musk one as far as I can tell is the labs cross check each other like pure of you. Like you do in science. Yeah. And then the Mark Zuckerberg one is please yourself. Right. What do you think about this middle one? That they should pace each other. Like, you know, evaluate each other. I mean, like I value you. I think like if we have boxing matches in the ring, the boxers should just be the judges of each other. Would that work? No, there was a screen foul all the time. Fowl foul foul. Like, you know, it's like, you know, the moment that the moment that the moment that the moment the other guy puts out the great model. And it's a big super intelligence risk. Absolutely. Like, you know, they have nothing responsible.

31:52Like, you know, when vested interests are at play and there's like IPO plans and these two companies are so competitive and they have like this history also between them. Yeah, they'll be very there. They'll be very fair to each other. I'm sure that's why you need to put it quite the right. And why do we have judges in the world at all? Why do we have third parties at all? Like, why can't just people like figure things out between themselves? So try if they want to do it, they should try. But I'm skeptical that they wouldn't just, you know, be biased in multiple ways to self like, you know, judge each other. So I have to ask, Golly, do you think something Elon also said, I think he was on the Olin summit. He was like, this is some elaborate for a DHS because on the one hand, you're saying all of humanity will die. On the other hand, you're saying, hey, what do you want for your IPO allocation? Right. And so, I mean, that is probably a more cynical view. But like, how do you, how do you reconcile that? I mean, the dissidents, I think, gets a lot of people. Like, how do you think that gets reconciled? Look, I think all of these things get mixed. Like, I think there are people that are freaked out.

32:52And I do think that the people are saying, like, hey, if there was regulation that would pace us, sorry to use the word, that would be good for us. Right. That would be good for us. But I also think that people have that set interest. Right. These things, like, you know, usually people figure out a way to always get all of these things to align in their, you know, harmonically in their head. So yeah, do I think that there has been a tendency in the past of, in general, using also marketing stunts by saying, you know, oh my god, this latest model is so good that I train. It's like unbelievable. It's like almost scaring me. And then the whole world, like, kind of starts focusing on it. Yeah, there's been that kind of marketing going on. Yeah. But at the same time, also, as I said, the time from CVE to actually weaponize exploit has like been going down from years down to like minutes now, just in like three, four years. So it's real. The cyber attack star will. And this is, but there's also a great marketing ploy to, you know, whenever you train a new model, make lots of noise around how much of a, you know, crazy risk it is to the world.

33:52It helps you, right? So, you know, maybe they're not in contradiction these things. So, I mean, you and I are networking folks. And there's a long history of forming third parties to help arbitrate things, right? Like IETF for, you know, I triple E or, you know, even like I can. I say it is gone. No, no, no, no, no. My question to you is like, so I think it's actually this is a very sensible proposal that they actually have. I actually agree with you. You probably want to make sure it's independent, which is not very right now. And there's going to be a lot of arguments over who you put there and everybody's mind disagreeing. Right. But you said, you know, why do we have judges? So that's like actually like the state stepping is actually quite a different thing than basically industry self-policing. So like at what point in time do you think it makes sense to actually consider federal involvement? Or do you think that now is a time to actually consider actual federal involvement as opposed to like more industry self-policing? Well, this is a very different. They are different, but they kind of bleed into each other. Like, you know, like Princess Findreill, you know, it's not like a completely independent self-portrait.

34:54It is, but you know, it's linked to the government. So I think these things like kind of will bleed over. Do you think that they evolve into, historically they've, they've, they've made the industry self-polices and then it evolves into regulatory. Look, if they are saying there is existential risk, which they're saying, you know, and they're saying, come police us and regulate us. I think it's very hard for regulators to say, no, we're not going to do that. So far they've said that, but I think that's not going to last very long. You know, I think it wasn't David Sacks was like, I've never had a CEO ask us to regulate that. And my favorite thing is the other, I've never had a regulator that says no to that. The reality is like the actual like medical machinery is actually in motion already, right? Like everyone has a talking point. Obamacans came out is a major issue. Do you think that there's a reality that is too late? This will be a major issue in the midterms and we're actually going to like heavy handed federal regulation. And this is all going to be paused, you know, and it goes into the, and throughout the coast into the DOE. And we're past that point or do you think we can actually end up with like a sensible self-policing regulation?

35:54I mean, I think we're just headlines you cannot. We should strive towards doing the right thing. We still have some degrees of freedom of how things evolve and there's still time. And yeah, you're right that largely you have these companies where we're pumping in so many billions of dollars. And the way reinforcement learning works is that you give it the reward function that's verifiable. Like we're going to solve this math problem or this kind of, you know, this narrow area of programming and so on. And we're pouring so much money into that, you can get quite good results in that narrow kind of, that doesn't mean that you're getting that super intelligent. You can even trick yourself into thinking that like less inputs are giving you a better outcome. Just because you're running so many experiences and thought about it so much, right? But it's actually very hard to do a closed, closed experiment this way, given how many resources you're going to. Well, the fundraisers are going up astronomically to your point. Yes, yes. And that's why these companies are going public, right? I think they would otherwise have, they would say probably, I mean, ask someone who runs a private company at scale. I think they would prefer to stay private otherwise. Yeah. Why are they going public? Because they need the capital and they consider the scaling laws and the capital to be strategic advantage.

36:58So that's why they're going public. But I would say let's go back to the four things that I listed. Yeah. If those are true, you know, if those four are true, would you want to be no but it? And is that would that be where some of that could get out of hands? Now there's no evidence that those four are happening. But if there was like, you know, no, that is actually where it's at it. Yeah. I actually think understanding for any system, like any sort of self propelling property is important. And we've done this in the past with dynamic systems, right? Like we've done this with like whatever a compiler did this with all the research on nanotechnology. Like it's been a common interest of ours. Yeah. And I don't think that that's new that it's an interest which you continue to have the interest. I just think the fear is that these particular systems are net economic systems that are so complex. Yeah. That the risk is is crying that you're seeing it when you're not seeing it. Yeah. And I think a lot of that's happening right now. Yeah. But I think of course if you see it, you know of course. I mean, you want to know. Yeah. But it is fair to say that the labs are now focusing a lot on RSI. And that's where they're headed next.

37:58And maybe they're just unjustifiably like word themselves. Just like they were word about GPT-2. So I was like, there were GPT-2 is world ending and then it was on to GPT-3 and for. So I don't want to quibble a lot of the times when they say RSI. They're actually talked about adult catalytic effects and adult catalytic effects have been our industry for a very, very long time. So for example, like there's no way you can create a computer chip without a computer chip. Yeah. It's like you cannot do it. Anyone with computer science agree. As you agree. My final right, it's my own buyer. Well, that becomes closer to RSI. Yeah. But like the steam engine was adult catalytic, right? So listen, my full time job is people coming out of labs in starting companies and they all say RSI because everybody says RSI. Like maybe 1% of those are RSI. Like they're more like we use AI for data cleaning. We use AI for making these channels. Let's make these things. So you're saying it makes basically catalyst in a sense that they're using AI. I just beat things off. It's auto catalytic. Yes. So I would say what we should be far aside. Like a model trends in other models, right?

38:58You're using a model to build a GPU kernel. You're using a model to do data cleaning. Just like I use a computer to set this. It's an auto catalytic. Which every tag, the internet was auto catalytic because it allowed people to collaborate remotely. So I would say, yeah, this is anecdotal. 90% of the calories are auto catalytic, which is 100% with you. And that's been going on for a while though. I mean, that's not even new. But I would say, but also now there is a focus on let's move towards actually can we get the model to train itself? This kind of like the auto research that Carpati did, but now I want to do that. There are teams doing that. It is not nearly as many calories as you would expect. And I just feel like I actually have a good sampling of this because they all come and talk to us. Right. So can we get all that data? I personally don't think it's like very high probability that those four criteria are happening. We're dropping, went on the pot this week. It says where did you get your air out? Yeah. So now I ask him question after he said that. Now everybody's saying we have a GI.

39:59So people follow what they say. But people have for a very long time when I asked this question said that AI smarter than most of people are on you most of the time. That's been almost like since Q3, Q4 last year they've been saying that. Then I asked them how many of you have hundreds or thousands of agents that you are managing that are coordinating with each other in swarms and negotiating and automating your life and everything around you. And if so raise your hand, it's like almost nobody raises their hand. Of course, Martin has done that. You know, most enterprises are on Microsoft co-pilot. Yeah. Like that's the extent of their AI. Most enterprises I talk to when I asked this question, they're like, no, we don't have any of that. So we're like, what are you doing then? They're using a chatbot. Like they're asking questions from a chatbot. That's basically very, very glorified, efficient Google search of the old day. And the results are just faster Google search. And their coding is happening. So people are using it for coding. Though the ROI is, you know, we can discuss that right there. But there's no like, agentic like work that's automated the whole enterprise. That has just like not happened.

41:00So then why is that? And I think that the real reason is if you actually look at it is, the models are smart enough. But they just don't have the context that exists inside of any organization. Like they have not been in every meeting. They don't know what's in everybody's heads. They don't know all the processes. They don't know. There's always like a couple of employees who know everything in an every organization. You know, you go tap on their shoulder and they're like, every is like, oh my god, what would happen if he or she quits. You know, they don't have that context. And if you just fuse that and gave that context into the AI models, just a frontier today, I think there's so much productivity gains you could get for any organization on the planet. For that, we actually don't need smarter models. So we don't need a smarter model that can actually solve Navier Stokes or conjectures or do better on humanity's last exam. Like we need it to just go from 60 to 70%. None of that is needed. So I think actually people are very upset on some like, oh, we pay different here. But actually if the frontier doesn't advance, it doesn't actually matter, I think, for vast majority of organizations on the planet. They're just so far behind in the adoption curve of actually automating things and getting value out of this stuff.

42:01But it would be disastrous to the labs because the price of intelligence is dropping asymptotically. I think it's going down by one tenth every six months or something like that. So that will dramatically change their businesses. It's like you weren't pushing the frontier. Yeah. But this is what we should focus on, right? We should focus on like, you know, there's like two sides of it. We discuss here a lot of their costs. Like there's cost benefit analysis that we should do on everything, right? We've discussed the costs a lot here. Like always there are like existential threats. There's their cyber risk. There's things we should be worried about and so on. That's like the cost side. What's the benefit? And I think now that this has become like a public thing and the whole public cares about AI, they're asking, hey, what's in it for me? What am I getting out of it? It seems nothing. So how do they get there? What are some of the use cases you've seen today? What are some of the use cases you've seen to date that have maybe surprised you to the upside? Yeah, I mean, first of all, there's like so much worry about existential risk and so on. So I think a lot of people just don't know what a cool use case is where people are actually doing interesting things. We have a lot of use cases that are, I mean, just fascinating.

43:03One that I like is crisis text line. So, you know, they actually use one language models with us to detect if teenagers want to do something. Yeah, wow, yeah. It's a class of awesome use case. And it actually saves lives. So that's a great company and that's, you know, that organization is doing amazing work. Another one that's kind of interesting is the Omni pod, which is for diabetes patients, they can put the Omni pod. And it uses a really really learn your insulin release and your glucose levels and actually exactly release. I don't know if you remember people used to like stick themselves, right? But this now happens automatically and it's like self learned AI for your body. It's a cool use case. Zip line is another one they're doing also. But when they started, it was like these drones that had, you know, they were completely automated, all AI driven, everything from the battery optimization to the routes and everything. And they were delivering food in areas of me.

44:03Blood to refugees. Blood to refugees. Yeah, started in Africa and then elsewhere in the world. So yeah, that's all, yeah, it's AI use case, you know, built on Databricks. So it's a cool one. But there's more advanced ones also. Like one that I kind of like, but it's hard to maybe explain is this model that we built transformer based model that we built with Merck. It's called Teddy transformer and has drug discovery. Yeah, so any public actually the research you can you can check it out. But it basically it's a model instead of predicting the next token in English. It predicts what the gene regulatory network did. You're in is going to respond and it can really detect, you know, which cells are causal and which ones are just reactive. So they're just reacting. And therefore they can start using this in drug discovery and get costs down significantly for developing drugs that are targeting specific diseases. So that's a super cool use case. There are lots of these, you know, Jeannie I mentioned. You have this ontology and you can ask any questions.

45:03No, no, this is using this. So you know, they built this GLP one drug. But what no, no, no, it's doing is now they're using it for all of their trials that they're running. And it can compress the time it takes to get insights. First of you're doing a BCD study or something from weeks down to minutes. So there are a lot of amazing use cases of AI. We should not forget these up sides also like we want all of these and we do not want to paste these. Exactly. You're totally right. And so how do they let's if you map out the next 12 months, how do the enterprise actually get value? You drop the word context, but how do they operationalize that? It's actually harder than most people believe, but you know, first and foremost, we have to make sure that we have digitized everything that's happening in an organization. That actually you cannot actually just, you know, have a magic wand and make that happen. So, you know, every meeting has to be transcribed. You know, as you have to be able to get all the context of all the meetings and everything that's happening, all the digital content has to be fed today.

46:06So you have to build, we call it an ontology, we build that. But first and foremost, you have to collect that. That itself is a problem in many organizations because legal teams will say, don't record a recall, don't record a free thing. So you have to do that in a way where. You can find ontology for everyone because I know Palantir says the word a lot, but it's not like they own the word on. Like what does that mean? And for the people listening, like, I mean, it's the, you know, ontology just means that in an organization, the relationship between all the abstract concepts of all the goals and all the departments and all the people and all the projects that are going on, what do they exactly mean and what's the relationship between them, the people, the resources and what that company does. So it's the difference between a person who is a new employee in the company and just started today and a person that has worked their five years. You know, let's say they're equally skilled, they have the same educational background, they're equally smart and hard working and all of that. But one, it's, you know, her first day today at work, the other one, she's been there five years. What's the difference between these two people?

47:08The one has an ontology of how that organization works, who the people are, how you get stuff done. Don't look at the org chart. That's don't go ask that person. Yeah, he will not get anything done. You go ask this person, you know, he'll get it done for you. And, and that's not how it works. You don't need to file that paperwork here. And, you know, and this is this project. This is what's going on. This is essential. So there's just a lot of ingrained knowledge that's sitting in everybody's heads who knows how an organization works. That's why it's people say in startup plan, they say, hey, if you lose most of your people, that company can't recover from it. You can't just replenish and hire new people like the people are so essential. How do we get that context? That's don'tology and give it today. Part of that is we just have to have, you know, the recording and all of that. But the second part is, how do you actually distill it down into a graph? Actually a digital graph that you can then feed today. I so the way a lot of the agents work today like a cloud code or any of them, codex or pi or, you know, open code, you can go through the whole school of them.

48:11You know, they have this loop, a genetic loop, it can reason, but then it goes and checks every resource one at a time. So we'll go to this MCP server for your question and try to see is the answer here. Is there another one it synthesizes it and gives you an answer. But it's kind of slow. I like in this too. If Google would have built Google search this way 25 years ago, we would have said, okay, we're going to get 10 blue links. We search for key terms here. But instead of giving you 10 blue links, it would have gone to one website, summarized with an LLM what it does found a few hyperlinks jumped in parallel to a few of them read a few websites done that for 10 minutes and then giving you like its best 10 blue links it would find. Well, that would be very expensive. Cost a lot of money to do that every time go on the web to it would have taken a long time. We'll get away to 10 minutes and three to quality would be bad because you're actually only looking at a very small subset of everything that exists out there. So how do they do it? They have an index. You never leave Google servers. You search for it hits the index the reverse index immediately gets you the 10 blue links within less than 100 milliseconds.

49:12What did you do the same thing for the yet? So the ontology is that we need to compute that index offline all the time. So it's almost like the page rank algorithm that Google had invented back in the day, but it's more complicated because Google was just looking at a web where everybody can go on the web. Here, their permissions, the links existed. Yeah, here are just permissions involved. The data I'm allowed to access might not be the data that you're allowed to access. So there's privacy. There's access control. Also, there's many different types of objects here that we're dealing with, not just websites. So the problem is a little bit harder, but it's manageable. You can actually do it. So, you know, the I'm convinced you can do this and you can get massive productivity gains out of it because we did it for the other. Yeah, we did it for ourselves. Yeah, and we're like the company just completely changed. It's like not the way it was. I would say a year ago because of this. I mean, you've been dog food and data bricks for data bricks forever, but maybe say more about the impact you've seen as an organization. Yeah, I mean, once we got this ontology and we started working on it and we actually have probably the largest of all of our customers.

50:17We have the largest ontology. Our ontology is bigger on us than any of our customers when they use us to build their ontology because data bricks uses data bricks more than anyone else uses data bricks. And so it's like millions of millions of nodes in the graph in the ontology graph that we have. So it's just, you know, what happens in an organization? What happens in an organization? You have a tree structure organization and information flows up and down the tree structure. You know, if you can't make a decision, you escalate your boss. Maybe they can tie great. It's a place up. Then if you get up to speed on what's happening and then you get all the context and then they make decisions once decisions get made, you have to percolate them down in the organization. A lot of this can now be done by AI if you have an ontology. Why? Because, you know, what happens in a meeting? In a meeting, you go through some, you know, someone has done the analysis. You probably have a PowerPoint deck with some pretty graphs in it. That person did the analysis is some smart person that used Excel, made some models. There's some numericals.

51:18So a lot of that you can just do with AI. So the AI can do the analysis for you. It has all the context. It can present it in a way that you want. You can ask questions about it. Instead of having follow-up meetings, you can directly ask questions directly from the AI. So it's very similar. It's along the lines of what Jack Dorsey has said that you can do the organization. It's just a concrete way of implementing it. So it's game changer for us. Like, you know, it's just everybody's on their phones now in the meetings on Genie. And they're like asking Genie questions. You can see, soon as someone says something complicated or something like, you see everybody go through the phone. Can you share that finance? Like the finance anecdote? You mentioned once in a board meeting. Yeah, it's, yeah, sure, internal board meeting. Yeah, so. Only a coach. Yeah, exactly. No, it's actually needed for for one of our presentations. I need to know how many customers do we have in Fortune 500? What's our penetration of Fortune 500? And I asked one of the people in sales ops because I thought she would have it. And she texted me back and said, oh, sorry, I can't log in to Genie right now. I'm on a flight.

52:19That's why if you're just going to log into Genie, I can do that. I asked you because I thought you had something alternative that I don't have access to. So then I was kind of a little bit angry. So I texted the CFO instead of Dave. And so the text that Dave said, hey, do you know what our Fortune 500 penetration is? And he just copypasted a screenshot of Genie back. So he also asked that. So does anyone do anything novel here that is just going to Genie and asking the ontology, you know, questions. It's like, let me Genie that for you. And so let me go. That's what everybody's doing. Now we just say, because I hate can someone just you need this? I think just get it from the ontology. So I do think it's a game changer. But it's not just you press about it and you have an ontology in an organization. And I think Palantir actually has done a great job of going to organizations and getting a lot of that tacit knowledge written down and getting it into the organizations. We automatically take that and build the graph. And then we feed that graph into the agents so that we can answer the question and answer it in a way that business leaders would like to see it, which is in graphs, you know, in a political way and a way where you can interrogate that question.

53:24And you know, continue asking questions and getting answers to those so you can make decisions and then disseminating that information and organization. Yeah, it's pretty amazing. You sort of bookmarked the developers are obviously using AI question will value. I want to follow up with you on that because I feel like you guys were one of the earliest. And I say, I don't want to use the word token maxing because it has such a negative connotation. But I think in terms of applauding people who can use AI to become more productive, you guys are going to go, you know, at the forefront of that. Right. And then of course, there's this cycle of oh shoot people are being wasteful. Now we need a value max. Like what was your your own journey on that and like how do you guys think about value maxing not token maxing and then I'm going to throw in unity gateway in this right because I think the managing of cost piece is actually getting more important and you guys are helping people do that. So maybe tie that in to extend it. Yeah. Yeah. So around Q4 last year was when you know the models got really, really good. And we started noticing that okay.

54:29It's actually starting to give much better productivity. So I actually started using models myself to sort of start, you know, commit code into production for later. But the actual as I want to take it all the way to production. So I did that and and started pushing the organization that hey, everyone needs to do that. I have done it. Why are you not like if the CEO can commit code to production and a very sensitive platform that has all these security requirements. You should be able to do that too. You being any manager, anyone in the organization. So started pushing everyone very hard and we started making leaderboards in Q4. And at the beginning of I say January, February, when kicked off the year, we were already full swing. Everybody was using the stuff and we were pushing and we're managing this. But you know, the whole token maxing thing was happening around you know February, March period already was happening. So yeah, we just had a lot of being maybe a few quarters ahead of folks to see what's happening here. And it was getting out of hand. So we already had a gateway. So it's called you in the gateway where we were already this gap was being used to provide token capacity. So you can get opening eye and through a pick, Gemini, GROC capacity.

55:35Like any customer can come to us and we'll just provide them the capacity because we have a relationship with those and any opens or small. So we started putting in budget constraints in place and giving people warnings like, okay, you have this much of your budget left. You're getting close to your kind of ceiling. So we started doing that per person and for group and then we started doing great analytics so we could predict exactly where the costs were going. And then yeah, and then we added smart routers that could actually pick cheaper models if you're getting close to your budget or if you know you have simple questions, we started doing that. We also built a harness called Omnigen, which can multiplex between the different harnesses turns out actually the harness itself matters. Like if you use the same model, the different harnesses, there's almost two X different cost difference. Even exactly same model, you know, same version, but different harness, you get two X difference in actual cost. So if you can change harness, you can get a lot of leverage in the cost. So we started using on this, that we were able to actually bend the curve and actually a cost for AI has been basically the tokens will continue to go up, but the cost has been sort of stagnant.

56:41So that's been actually super, super important for us. And there's huge demand for this. I think every organization is going through this now. Yeah, I, yeah, for the first time, I do a lot of board meeting someone 20 something boards for the first time ever company at scale last week said that they're moving from the frontier models to GLM, this is a large engineering organization. Do you see this? Do you think that that's a trend? Do you think that's just like a one off anecdote? Because I've been hearing about remember the first deep seek moment in the video, start driving and that turned out to not be real. Yeah, then the kidney moment and the next deep seek moment, none of it seems to have actually had an appreciable impact on the market. But now the amount of anecdotes that I have are pretty real and this seems to be happening. Yeah, I love your view. I mean, I think people want both. They want, you know, they want the latest model that super intelligence for the difficult task where they get ROI. But then there's a lot of mundane, dumb things like, you know, you, people literally use their harness to rename files and whatnot. Yeah, you know, like it's your pain, you know, orders of magnitude more for that.

57:43Please type that in yourself. Don't have the model do that. It's going to spin for five minutes and then it's going to rename the file for you and cost you, you know, sense. But I guess so people have just wanted to actually see more. Yeah, no people moving on it, but what they're doing is that, you know, the pattern is either use, you know, you can, you can use this expert pattern where you have, you know, small cheap, broken source model that uses an expert model, the big ones or vice versa or a way in which they can sort of ping pong them to each other. But also multiplexing harnesses and just changing harnesses so you can control the costs is also what people are doing. You know, people have found for instance, you know, there's pie is very efficient when it comes to as a harness. So yeah, I think there's going to be a multitude of these. It's easy to the models themselves are stochastic, as you said, every time they give you different answer and they're changing so much. So there's just a lot of experimentation happening. So I think we're going to get to a world where you're not always using the smartest model for everything, which is kind of the, being the paradigm for the last couple of years. Like new model comes out. It's super smart. They use it for everything, even really, really simple mundane tasks.

58:46I'll tell you what I see. I see people using Fable and Astra for like architecture, a cheap model for implementation and then Fable or Astra for audit. Yeah. That seems to be like this emerging. What are you guys seeing in the startup? I mean, aren't they? That's it. That's it. That's that's honestly the pattern. How much open source? By token and by dollar either. So by dollar, open source is like 5% is very little, but by token count is over 60%. Yeah, I was going to say, I mean, we talked to let's say a Decagon or something like that. They well, I think it's different internal use versus external for product on the external for product. I think they're almost up to 90% open source on the internal. And I don't want to say for Decagon in particular, but a lot of them are like, we don't care. We'll just use frontier. We're not thinking about cost control. But as it gets bigger, right? You and I are on other board meeting where they actually didn't bring that down just from a waste perspective. So I definitely see that moving more toward open source on the product side. And actually that's a related to another question.

59:51Maybe around open source, but post trainings, specifically. I feel like you were kind of early. I remember talking in 2023. What did you buy? Bozac. 2023. So this vision that you had in 23 kind of came true in 2026. I don't know if you guys would agree. Right. Like that's sort of what we're hearing across. You know, of course, oh, just sort of like, hey, we're going to actually you're going to own your own intelligence. You're going to be, you know, open source models, etc. And that's definitely what the startups are doing. I don't know if that's what the enterprises are doing yet. But like, I mean, do you feel like you were early to that? Yeah, I mean, first of all, you know, when we started, it was also, hey, we're also pre-training for you, which that doesn't make sense. You know, you can, they're so the very good pre-trained model that it can use, but that you can do actually post training on the model and you can do reinforcement learning. Yeah, we actually doing it that scale. And many of those startups are actually customers. So we actually helped them, you know, using early or reinforcement learning environments where we can make the models very, very good at the specific tasks that they are doing.

1:00:53It makes a lot of sense for them to do that. If you have a repetitive task, so if you have a startup and it's offering a product and a product does something specific. It's not just a general intelligence. It does something specific for you. It makes just a lot of sense to take a really good open source model and, you know, use reinforcement learning and make it really good at that specific task. You can cut the cost down. You can make it really fast. You know, the controller on IP. So in that sense, that is possible. But large enterprises, they just need basic automation. And it's just too much for them to do this right now. I think one of the challenges is, you know, you need good devals and making good devals is hard. So while the startups can do that, then they're motivated to do that. Other organizations, the easy button might be just to use a frontier model, then having to create your own evals. We actually generated even, you know, you've asked for the customer automatically in the product and we had it front and center. But then people want to use it. So it's okay. Let's move it to the back end so that it's optional. And then they would never go to it.

1:01:54So I would say in general, why they just don't want to get into it's too complicated or I think you want quick, you know, quick reinforcement of like, you know, hey, there's a new model. I want to try this out. I want to get this problem solved. You don't have time to go do this. The scientific method of let's make an evil. Let's have a great baseline. And it's sort of like TDD test driven development. You know, it's off during the day and just people actually do test driven development. Very few did right. Everyone said it's the right way to do it. But nobody actually practiced it. So that's the same. That's kind of a little bit of the curse of, you know, doing training your own model is the evals is the hard part. I know you you have an NFT model at data bricks. That's a very popular word right now or acronym. But does it like to get these enterprises that large. Is it a full FD model that's required or like how do you and how's that evolved maybe? Yeah. Yeah. I mean, we've had these FDs and it's the demand for it's gone up significantly. A lot of it is, you know, how do we build that ontology? Like the ontology is automatic. But if you're not collecting any information, like you're not recording anything. Right.

1:03:01So that's one of the key things that we do. But also things like, you know, I want to go to an agent. I want to put it on, you know, I wanted to be customer facing and it's a really low latency. And I wanted to have guard rails to not people coming to abuse it or ask it things that we don't want it to answer and so on. So we can build that like, you know, like sports AI that Fox has you can watch out with it about sport events. You can try to ask it actually about politics and it's very good at rejecting you and talking about sports instead. So, so the FDs built that. So, you know, we'll help the organizations actually get started with AI. It is important because it's just many organizations do not have the in-house expertise to build the stuff. So they're just a little bit of help on the side. And then they get started. So, yeah, makes sense. So this is more related on the agent side. But I saw recently that I think a third party, neutral third party. I think did some tests that. Lake base or neon was actually the data the postgres database of choice for agents. And I thought that was interesting one because, you know, one exciting data works. But two, I probably wouldn't have guessed that maybe a year ago. Yeah, I was a surprise.

1:04:10Right. Just because there's others out there that have, you know, great developer momentum as well. But it was pretty clearly number one. And so I'm curious. How did you guys crack this and what. What makes you win across the agents because you win the agents now you win the market. Yeah. I mean, I think a lot of credit should go to neon and guitar and team. And I think what they've done is they've just been obsessive about how do you make the models. How do you make the models pick an agent favor lake base or neon as a database. So what did they do? The agents want experiment, you know, they're going off. They're trying to build a little bit software. They need the database. So you need the database to come up quickly. So they had this obsession that everything should take less than a, you know, far less than a second. So you know, database comes up in far less than a second. You can clone gigantic database. I can't wait about database. You can clone it in less than a second. You know, so it's like highly elastic, highly responsive. And then they built this killer feature called branching. So branching just lets you branch the database. You can have many, many branches over the same database. And they just made this very, very lightweight. We saw this with other things with agents, right? Like UV, you know, rip graph.

1:05:23Like basically re implementation of a lot of the tools on Unix, making them really, really blazing fast and lightweight. And also sort of fail safe for agents. They just did this to a harder problem, which is database. So like now you have a Postgres database. And the Postgres rate risk has all these advantages that it's really fast. It's nimble. It's fail safe. You can, you know, go back to snapshots. You can do those things. So I think that's why it's just easier for agents to use this. They also make sure that I had a pricing model that was like, you don't want just because the agents are building some software and experiment. You don't want the cost to run out. Yeah. You're okay paying for your database. If it's like production using lots of people are using it, but just experiment. So I think they were just obsessed. They were not trying to win the database war or trying to be better than some other vendor. They were obsessed with how are we the best for agents? And that's a new persona because in databases, the obsession has been how do we help TBAs? How do we help, how do we help help the people that are using the database? They changed the game and said, hey, how do we focus on agents and help agents get the best database they want.

1:06:27And you know, now over 90% of their databases that are created on neon and like base are actually created by agents. So it's not humans. So, you know, numbers to create themselves. They're, by the way, it's remarkable. So I've started using neon as like my standard database. And it was bizarre to me because normally when you enter a large company thing slow down is actually like the product has got materially better. Yeah. Are they totally independent? Do they work with the rest of the like? No, it's a great team. I mean, I'm working very fast with it together. We're very fast together. You know, we love databases and data. So it's, you know, we live that. But the team doesn't great job of just making super fast snappy and great for agents. All right, Ali, what is your P doom? Less than 10%. No. Close to zero. What about yours? I don't know. I just say my, my only answer is my P doom without AI as much hair than my P doom would say. Wow.

1:07:27That's not what I was saying. I went on a technical. I would agree with Ali and this one. Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review and share it with your friends and family. Or more episodes go to YouTube, Apple podcasts and Spotify. Follow us on X, A16Z and subscribe to our substack at a16z.substack.com. Thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.