TNO071: The Network Team Is Drowning. Is AI the Life Raft? (Sponsored)

0:00[♪ OUTRO MUSIC PLAYING [♪ Welcome to Total Network Operations. Our mission is simple to bring out great ideas in modern net ops and networking. I'm your friendly neighborhood podcat host, Scott Robon. And today I have the great pleasure of welcoming Rico Chenoy and her fun Kim G of Backbox. I'm going to ask them to introduce themselves in a second, but let me just give some setup for today's conversation, some context. The last year, 49,000 CVEs were published. That's more than 130 new vulnerabilities every day. Do you feel overwhelmed? I feel overwhelmed by these numbers. Did your team grow by 130 people? I don't think so. So how are all these CVEs getting addressed in your network infrastructure? Well, today, Rico and Irfan will join us to discuss why the math of manual network operations no longer works, keeping up with security upgrades for infrastructure and more.

1:00Where traditional tools fall short. And whether AI-powered automation can offer network teams away forward here. So let me welcome you both. And Rico, can I ask you to introduce yourself in Backbox first, and then we'll go to Irfan. Sure. I'm Rico Chenoy, CEO of Backbox. Very excited to be with you, Scott, and with our broader community networking. I've been in the enterprise software security and networking space for the last 25 years. And I couldn't find a more exciting time to be in the space. But also, as practitioners in the space, I have seen every kind of change. And there's a lot of, I told you so, happening that AI is kind of challenging us with today. So exciting times to be here, but also an opportunity to really change the game, if you may, for network practitioners. Very good, very fine. Yeah, so I'm the field CTO at Backbox. Effectively, what that means is I've seen a lot of things,

2:03some good, some bad, some ugly. I started out as a vulnerability management specialist. So I've been on both sides of the coin, building and running vulnerability management programs, and then helping infrastructure teams remediate those vulnerabilities. And really getting to understand, what does it mean when we're faced with so many vulnerabilities? And how do we prioritize that? What do we do? How do we fix it? A lot of times organizations and people are like, hey, I can't fix this. Well, let's unpack why. And what's the use case behind that? What's the reasoning? And so I've spent a lot of time learning from infrastructure professionals around what's challenging. How do we overcome those challenges? And hoping to share some of that today with the audience, so looking forward to it. Definitely look forward to what you both have to say here. And Rick, I agree with you. This is the most exciting time in my career with everything that's coming to market and the challenges that we have in front of us. Let's dive in a bit.

3:04Let's start with the math. We threw some big numbers out there at the beginning. And we have this new entrant under the scene, AI models finding more and more vulnerabilities and day zero issues from the start. Do you expect those numbers to go down anytime soon? Well, the question is how exponential is that growth, not whether it's going down. And so it's not a theoretical exercise at all anymore. When we talk to our customers, it's dramatic. How it's changed things for them. So a couple of things about the 49,000 CBEs. When we've talked about vulnerability management for years and years and years, the thing that people assume very quietly is they're patching Windows MacBooks Linux. So they're thinking servers and endpoints. But in reality, the part that's really being very actively compromised is network infrastructure.

4:07So it's a small portion of that, but it's actually the most actively compromised because it's easier to do that than it is to do complex. They don't need to do complicated things on servers anymore. So that's point one. The second point is as much as they're being compromised when you think about vulnerability management and patching exercises, it's that much harder to do it on network infrastructure because it's not like deploying a Windows patch across thousands of servers, not to say that's not complicated. But that's a relatively vanilla exercise compared to just the disparate systems we're talking about. The hundreds of device types alone and vendors and so on. The complexity begins when that vulnerability management report runs and it lands on a network infrastructure person's desk on a net ops desk. That's when the work starts. And to your point, no, we're not getting 130 more people.

5:07No, they don't spend all their days just trying to figure out what is this vulnerability and how do I patch it. So the problem was where you had maybe quarterly patching strategies if you were advanced, maybe annual strategies, which may be what a lot of people are doing, it's just not enough. So yeah, the problem became worse and it's carrying how much manual labor is still very much the biggest challenge there. You see, you're bringing new light on this to me. And even as somebody who's been in the industry for decades, that's a variety of operating systems. And even if they're from a single vendor, some vendors have multiple trains of operating systems on different devices. We won't point to anyone in particular here, but the audience can figure out who we're probably talking about. And it's not just one, right? And also like the blatant lack of hot fixing, right? It really is an OS upgrade when you find a vulnerability. Different vendors have tried different things on the hot fix front, special SPUs and other smooths

6:09for certain issues. In-service software upgrades, never really living up to expectations and truly staying in service. The IS of ISU is often disappointing, let's leave it with that. So yeah, you're right to call out the complexity in the network infrastructure side. Refond, I wanted to make sure you had, if you had any comments on this too, please chime in. Yeah, the AI point you mentioned earlier, one of the big things, it's not just about, mining the vulnerabilities, but it's then building exploit as well. So if you think about it, historically, many organizations have just accepted the risk of network-device vulnerabilities because it would take months, years, like Rekko was saying to patch devices, and one customer I talked to, used to take three years to patch their devices. They would do a full refresh every few years. So I was talking to the CISO and I said, you're doing this, you've got a patch management program for your workstations, your servers. How are you handling network devices?

7:09So honestly, is the risk really all that? Because one, it takes them forever to do it. So if I'm not an effort, it's going to take the team versus the exploitability. People are writing exploits, attackers are writing exploits for servers, writing it for web pages, writing it for, to gain identity access, email access, places where people are touching. And that was true, and it still continues to be the main point of entry. But if you look at some of this year's Black Hat talks on network infrastructure in particular, all of a sudden there's more exploitation happening on network devices because it's being left alone. Yeah, you have AI coming in and saying, well, hey, I can actually get in through this. AI helps you do things faster. And not just saying, hey, go trust the AI, I'm saying the AI is a tool to make things go faster. So people who are writing exploit code and trying to penetration test these devices, all of a sudden they're able to do that faster and develop exploits faster on areas in the infrastructure all our network that is previously just left alone.

8:13So it's becoming a bigger problem and it's something that we need to address as an industry sooner rather than later. Well, you certainly have raised the manual approach issues here and the environment that you just picture painted for us, you can't do this manually. You haven't been able to do it manually for a long time, even before AI came on the scene. What are you seeing with different ed ops teams? Like, do they welcome you with open arms when you come in with Backbox and say, thank you or do they need some education on, look this is worse than you think? What does that look like for teams really engaged in manual operations for security upgrades? Yeah, I think the first thing is just going in and speaking to people. So the last sort of 14 months have done a lot of asking network operators, how are you handling this? What are you doing? Do you realize it's a challenge? I mean, a lot of times people are just used to doing things a certain way and don't realize that there could be improvements made.

9:16And so the good thing is, there is a realization happening with that. I talked to one, very large, I'll give you two examples. One was a large financial institution. They have about 20, 23,000-ish network devices or about 18,000 switches. And their vulnerability team, it's quite interesting because they're, they're see so managed to get a bonus structure to all the infrastructure VPs that if you maintain your risk score below a certain threshold, you get your bonus at the end of the year. Otherwise, there's complications. So when you put money behind it, it's a big deal. And so that team, the networking team in particular, they're like, how are we going to keep up with this? So they put in a business case. They said, we need 23 FTEs to keep up with this. 23. Now they have, they're not totally global operations and they've got operations in Canada, US, and Mexico. Try to find 23 people in three cities or three countries like that, let alone, even if the head count was approved, which it wasn't.

10:16That's a challenge. And then if you look on like the smaller scale, talking to some customers who were, there was one customer in particular that has about 2,000 network devices. And they are required to make two changes per day to a couple of configurations on every single switch. And that was costing them about $8 million a year. So they didn't just identify there was a problem, they put hard numbers behind it. This is how long it's taking as this is what the dollar effort is. And they're like, well, we got to do something to save this money and save this time. And in certain cases, you know, filling gaps where we can't even fill them. You must see some interesting workflow issues or even lack of thought around workflow for doing these things. Like before you cannot have made any of this, and I don't want to jump ahead, you have to understand what it is that you need to do. And it seems to me like this is a scenario for creating extremely overwhelmed network engineers

11:18with the volume of changes that need to be processed just not time of day to get everything done and bring enough personnel alongside your 23 number. I think that's probably on top of, look, we've had these two headcount open for two new staff members for a year. We haven't even able to fill those. Even if I could find 23 more, that would bring the total up to 25 in that scenario. So how often do you run into scenarios like that? Well, let me jump in for a second, because the way it looks from a network operations perspective is also interesting because to your point, the work starts when that vulnerability management product runs and a report lands on their desk. So to their point, how did they come up with 23? They just did the math of what all that work is when they get started, right? And so the issue is one of those,

12:19I was saying, I told you so network operators have been saying this for years. This is not sustainable, we can't do it. Now, the answer was, it's not important, don't do it. Do it once in three years and so on and so forth. That's what changed in today's world. So two things, one is why did it become a bonus situation? It's because the hacks became real, the attacks became real, they were either it happened to them or it happened to their competitors and someone on the business side said, this thing that we didn't do on a regular basis, we need to do now. So that's an interesting point. And then the second point is, so why is it so painful today? Because in addition to it's a bonus thing and we have to do it regularly, guess what? It's in the news. Glasswing is in the news, methods is in the news. And what that does is those business owners are not even on the IT side come running in and saying, do we have this problem? I know we have some Cisco's. Do we have this problem? So now that's work, right?

13:20A whole bunch of people need to say how much of this is our problem? How many of those do we have? How many of them supply and all of that? So at that point, I haven't done any patching, but I have spent all day answering those questions. So that's where the 23 comes in. So when you think about the manual work, rationalizing what the latest thing is, is manual work and that is thousands and thousands of pages of vendor information that network engineers stop doing strategic work and they go do that all day long. And this is expected to continue to grow and expand. A lot of those things don't necessarily involve patches because network engineers will tell you that they're not in the business of just patching systems. Their job is to keep systems up and running. They also have SLA responsibilities. So if they're able to figure out that is certain latest new thing that came from a business, a person saying he does this thing apply to us, their ability to say, yeah, we're already mitigated.

14:22You know, not spend the hours to figure that out, but to actually have something like that at their fingertips. Their ability to say we have, you know, work rounds, we have configurations and if configured properly, they don't apply to us. All of that stuff just to keep systems up and running and moving forward. That's massively reducing the workload right there. And those are the kinds of things that people are saying, I just don't have the ability to keep up with this. And then of course, there's patching and upgrades too. So when you look at just this sheer magnitude of what vulnerability management and upgrade takes, that's where the, you know, the, I would say, the last straw fell on the camel's back at this point in time. And we have more and more people saying, we got to find a solution. And so we typically are response to your question is I didn't know that something like this existed. Very good. Whilst you do want to talk a little bit about, okay, what are you bringing to the table in comparison to traditional tools

15:23that aren't as accomplishing what you're, what you're bringing to the market today? Yeah, I can take that one. So, you know, how do we solve this? I guess is where we want to get to. Right? We've kind of exhausted the challenge. And when you talk to, again, network, network engineers and automation specialists, especially, you're like, well, we got to screw the way to do this, right? There's many coding languages, you know, again, looking at the network automation form and some of the talks. There's a lot of, how do I build a set of code to run this type of automation? Because the actual task itself is not, not crazy, right? You need to connect the device. You need to stage the upgrade, verify the configurations, correct, run the upgrade, verify that it ran, you know, do your post configuration check, make sure you're still compliant with the standard, et cetera, and then disconnect. And so the structure generally is the same. So I could create, you know, some code

16:23and create a script to run it. I could vibe code it now with AI, and you know, come up with that. And, you know, if you're someone like me, I did my degree in cybersecurity. I barely scraped by in my coding classes. I didn't like it at all. You know, I did a bunch of networking, a bunch of security, and I just, I could not do coding. I can read the code, I can tell you what it does, but I couldn't write code from scratch. And just in speaking to a lot of network professionals, they're in a similar boat, right? Very much so. Yes, if you take the time to learn it, it's not overly scary. But there's gotta be another way. And so what we're bringing to the table is that other way to do that. And that's, you know, you can build these things, with that shell, and just plug in the command that you want in a load of no code environment, and then be able to run automations at scale, whether that's configuration changes in all your devices or upgrades, and be able to, you know, back up, test, verify, you know, back out of your fields, et cetera, and run all those kind of different checks.

17:25And I think the most complicated one I saw was speaking to one organization. They had written, now by the time they took into account all the different varieties of devices in their environment, it turned out to be 80,000 lines of code. And, you know, great for some people, the guy who wrote that got promoted, you know, built the entire team where I'm learning to code Python. And it sure is great if you like that. But sort of, I mean, 90% of the people I've talked to don't want to do that. They're looking for a better way to do it. I think that is a common, a common theme, you know, hey, I'm a network engineer or I'm a cyber engineer, because I didn't want to go and learn to be a coder, right? I like the tech, but I have this very different view of things. And I'm very sympathetic with that perspective. I do think, like I think about carpentry skills around my house. I'm never going to claim to be a carpenter. But I could swing a hammer, I can solve wood, to do functional things around my home.

18:27I've acquired some skills that helped me accomplish some objectives without changing my identity or my career. And that's one way I try to encourage folks in network engineering. You know, Python doesn't have to be that scary. Even Pearl doesn't have to be that scary. It doesn't mean we're trying to change who you are, but you should be upscaling on, you know, what else can you do to help make your job easier? And some people gravitate to that and some people don't. Yeah, even with, if you try to fly code it, right? One of the principles we talk about when you're building something with AI is at the end of the day, AI is a tool in your toolbox. You have to be the expert. So you could say, you know, hey, Claude or Chad GBT or whatever, write the script for me. Sure. It could look right. But if you don't know, if you're not able to read what it spits out, admit it on, you got to level up your skill to no weight or influence of that output. So when it comes in, before you go running that in production, please tell me if it test environment.

19:28If you go in and test it and verify that it works, because at the end, if it's, if that, you know, AI took that from, you know, source where there was mistakes or, or tried to fill in a gap with the hallucination or something, you could run into some challenges if you yourself are not the expert. So, so highly, highly encouraged, you know, folks to before doing that, learn a little bit about it yourself. So when you're getting that, that agent to build something for you, you can verify it. Sure. I also want to pull on the thread a little bit of just the alerting problem. And we've all heard the term alert fatigue, you know, anybody who's in that ops or psych ops, right? And that alert fatigue is increasing. And I think you both touched on this a little bit already, but you know, the difference between, here's an alert versus does it require a patch? Is there, you know, interlular mediation or work-or-ounds I can put in place? Like at a couple more lines to this firewall filter, this ACL, et cetera.

20:30That in and of itself takes work too. And it seems like that's part of the problem that you're definitely calling out. Yeah, and I think there's one more piece that I'm here, right? So where Irfan was going was, I think vendors like us, we've also done a bit of a disservice for our customers because we're pushing this AI, AI is going to solve World Hunger kind of thing. And so are their business owners, right? Who are not IT experts necessarily saying simplistic things like let's have some AI and save some people, right? Some headcount. It's over, it's over the top. The thing that we hear from network engineers to the point of alert fatigue and all of that is AI can be part of that problem, that noise because nobody I've talked to so far, who's responsible for keeping mission critical systems up and running, once AI to just go bump in the night. Nobody wants it to be agents that are automatically doing things. That's the scariest thing that you can possibly offer

21:31to a typical network engineer who's trying to keep things up and running because that thing that went off in self-healed and fixed something and all of that is now generating a whole bunch of alerts. And guess what? AI doesn't feel accountability. Who are you going to fire when that happens? And who's going to actually go and figure it out and fix it, right? So that's where we think that what people are asking for is responsible AI. Give me the smart intern to do some of this basic grunt work and give me control. So give me more ways to do some of those things faster, but also don't go making changes in the middle of the night or whatever. And don't auto, the scariest thing is self-healing. So when we think about AI, the key here is what are those things that enable a really strong smart network operations engineer? What gives them hundreds of hours back? What are those types of things that AI can do for them?

22:31So to your point about alert fatigue, if a new vulnerability comes in, can you automatically disposition it for me? Tell me how this applies to me very accurately. Tell me whether I'm automatically mitigated. Tell me if what this vendor is asking for, are there work grounds, are there patches? Am I already on that patch version that they're recommending? And therefore take off my plate, all the things I don't have to do and give me the five things I do have to do. And for the five things that I have to do, create an automation and let me go test it. That is control versus something that's going bump in the night. So we are finding that a lot of people say, hey, AI sounds scary to us, but there aren't a lot of vendors saying, hey, but there's a responsible way to use AI. I definitely want to tip my hat to you on bringing up this point. I agree with you 100%. And as someone who has spent a long time on the vendor side of the house,

23:31I'll be a pre-AI, hey, I've got this feature, you should use it, right? Well, take the time to understand, what does the customer really need, right? And now being very community focused and driven in some of the network automation forum, US Networking User Association and other public forum that I'm part of, people are hungry for real solutions, but they want to be consulted and they want to be part of the process of building trust in tools and their implementation. And I think that's a point that we can all do better on as an industry vendor and consumer and system integrator alike, right? What does it mean to build trust around a new technology before I deploy it in production? That's not just an AI thing. We've always wanted to, first here with the vendor has to say, bring it into the lab and kick the tires,

24:31maybe do an extended Pock or even a early deployment. Before we say, okay, this is ready for production in these scenarios and then grow the footprint over time. It's never a switch that just gets flipped. We bought it and now we trust it. There's gotta be a process there. I'm sure you're running into discussions like this with your customers. Yeah, Bingo, I mean, the first thing I said when you know, if you wind like a whole month ago, said, you know, everyone's talking about AI, you know, whether it's an investor or business leader or you know, it's generally the high level conversation and then the folks at the ground are like, well, what do I do with this? But it's, you know, the first thing I said was we're not building other chatbot. Okay, I'm sick of those chatbots. They never work properly. But exactly to your point and you know, shame this plug for an article I wrote not too long ago. We talked about, if you look back to cloud computing, right? There was, you know, initial pickup with cloud and then there was, you know,

25:32some things people didn't necessarily consider or some financial impacts. There was different, you know, with infrastructure and was it services of cloud with, you know, infrastructure as a service or, you know, platform as a service, whatever that case may be. And if you look at AI, it's kind of taking a similar trend where there's things we didn't necessarily think of, right, there's different flavors, different language models, different outputs, different consequences. And it's taking a very similar approach. Now, if you look, if we learn from history because history repeats itself, who, what made cloud successful? And the ones who paused and thought about what's the business outcome I'm trying to achieve? So, and those are the ones who used it successfully. Now, with AI, networking and everything, we're starting to see more on-premise infrastructure, but we can take those lessons and apply it to AI and say, you know, what are the business outcomes I'm trying to achieve? So if I'm a network operator or a network administrator engineer

26:32and I want to leverage AI, well, okay, for what? And if it's by code something, well, I need to understand that code. If it's building an automation to do something, well, I know as an network professional, I know what a connect command looks like. I know what a configuration command looks like and what a showrun looks like. I know what, I may or may not know RegX, but I'll know how to upgrade a device. I'll know how to learn a compliance check. And so I could say, hey, AI, build this for me. But one catch, the sources that that language model's using has to be the correct inputs. Otherwise, you kind of end up with garbage and garbage out. And so one of the, again, founding principles as we were building our platform here to be able to automate a lot of these functionality leveraging AI is we have to make sure we're taking the right sources. And part of that, one of the main principles we follow is transparency.

27:32And so being able to say, here's what I got and this is where I got it from. And not hallucinating, not falsifying, hallucinating even the reference, like this case didn't exist. I've seen that in some public models. But being able to say, if I'm building a Cisco automation, for example, I know the source that I'm using is a set of Cisco commands and Cisco automations that work that are there that have everything I need. And then it builds it based off of that. So if I say I want to run, you know, confitee, whatever, build me that framework, you know, you've got like a burger, right? The command is the meat of what you're trying to do. And you know, just give me the buttons, toast them and add some ketchup. And we'll make me that sandwich that I can now run in my production environment after having tested and verified. But it's from a trusted source, focusing on a business outcome. And then you're able to have that output

28:33that achieves a goal to make your business successful. Totally agree on everything we do needs to be tied to business objectives or mission objectives just to put it in the government context. They do a lot of government support work. And what is the problem you're trying to solve as a great clarifying question that should tie to your business or mission objectives for sure? And sorry, if I get that in one more thing, you know, as we were again, serving a lot of customers and prospects and people in the industry, we kind of found people fell into one of two buckets and those two buckets are kind of converging now a little bit. But it was, hey, I don't want AI right now. Like, look me alone with it. And it was, give me everything AI. And it's slowly merging a little bit. But when you drop the term AI for a second, which some people love or hate, and you focus on those business outcomes and say, hey, what if I could help you with that? You don't need 23 FTEs anymore. You don't need eight million dollars anymore per year.

29:35You're able to provide valuable business outcomes. Don't mention AI yet. Immediately that from both sets of those groups was, yeah, that really would help my business or help my mission or help what I'm trying to achieve. Okay, well, how do we do that? And now let's talk about the details of how to do that. But again, focusing on those outcomes is super, super key. I have this, I have a dream. It might be a little bit of a naive view that eventually stopped putting so much emphasis on AI as the mechanism and moves it back to exactly what you just said. What are the outcomes that you want? These are the, this is the what? What do you my need? The how is a separate issue from the what? And I love the fact that systems are emerging today to have the combination of I can use non-deterministic tools from AI for certain pieces of the puzzle. And I can use deterministic mechanisms

30:36for other pieces of the puzzle. And emergent systems will use the right tools for the right pieces of the job. I don't think we're gonna get away from the focus on AI anytime soon. But I'll be here waiting patiently. I'll be ready when we can move that back to the background and put more focus on actually solving the problem. But now you've both, I've done a great, great job of bringing out, we believe in responsible use of AI. We talked about some of the scary stuff, but there's obviously some things that you are doing where you're finding this useful and how you're interacting with your customers. So where do you see AI changing the equation for you on a daily basis? Like take us from that, the two parties discussion or fun and say, okay, you've put it in the context of, well, if I could help you with this and not say those two letters, do you see progress on there and AI really moving the ball down the field? Yeah, it's being able to demonstrate that outcome, right?

31:38I'm just talking about it, but showing what that looks like. If we again, go back to the vulnerability example, just something simple as, hey, I have multiple network vendors in my environment. I have a vulnerability report, I need to figure out what to do. And if you got a Cisco's page, Ford Nets page, Juniper's page, they're all different formats of data, everything's disappearing. And just that alone would take one person hours, days, weeks to go in depending on how many different vendors you want to figure out, what do I do and how do I fix these vulnerabilities? Well, what if you could just have AI scan those pages normal as the data and say, cheers, what you need to do? In that alone, right? Again, don't mention AI, but if I had a way to give you all those data at your fingertips, and then, oh, by the way, I can click a button and create an automation to remediate or mitigate if there's mitigating configuration factors, right from the vendor web pages. So I have legitimate data

32:38and then an automation framework to build that, all at the touch of a button. Again, AI is there to do things faster. Hey, you can do that. And when you show that outcome, and just being demonstrated and show it, you get a lot of, whoa, that was cool. And that's typically the reaction we get when we're able to show that. Yeah, we try to think about it in two ways. We can help our network engineers make better decisions faster. That's one area. And then the second one is to just make life better for them. And I think those are step function, massive improvement when we're able to do that. So for those that are on one end, where they're afraid of AI, what we start with is how do we give you at your fingertips the smart intern who helps you make better decisions. So they're giving you insights that would take you hundreds of hours to go figure out. Are those accurate and can we show you that?

33:39Then it's recommending things to you, saying things like, hey, we noticed that you were worried about HTTP enablement. You know, and it turns out you were looking at it on the Cisco device, but these three aruba's over here also have it. Should you be looking at it, right? Like making pragmatic recommendations based on what you're doing. That is super cool, because both of those help you make better decisions without going off and, you know, doing things in your environment. And it's not this far cry of self healing madness, which, you know, loses engineers. And supposedly it's a full network engineer. That's still an unknown to that side of the fence. The other side of the fence where they're like, hey, give me more. They absolutely love how fast they can go support multi-bender networks with automations that they just thought of, right? So across the board, I think they're sort of a take your own journey, drive it, your pace. Can we help you make better decisions and can we make your life better? Those are the two areas in which people start saying,

34:41yeah, this doesn't matter that there's AI under the covers. These are really pragmatic solutions. You're keeping the human in control, right? You're making sure that, you know, nothing goes off and does self healing by itself. Although I would say that's part of my dream state, right? We do like self healing. After we've, we know that we can trust it. And that does not happen automatically, you know, without testing it, without proof over time, for sure. So, yeah, go ahead, everyone. I got a funny story about that. We a decade ago, I worked at a company called Tripwire and used to be able to track changes on devices. And then also in front of that script, when something happened. And we had a, it was back in the website, the Facebook era where somebody tried to deface the website. And so there was a change that happened. But we had it, you know, that page had to be static. So we had a script run anytime, that page changed, just changed it back with a known good copy.

35:42And this was pre-AI, pre-all that stuff. And we only detected there was an attack because the page kept switching. And we were like, why is this? Switching back and forth. Oh, there was this tech going on. We should probably stop that. And so that, you can do that in static areas. Or when you're able to put guard rails today around what the AI can do. Hey, AI, you're allowed to do this one thing. I know, you know, Telnet should never be on these devices. If you need to tell that on, turn it off. I'm sure it's what turned one on. Yeah, imagine there's still some of those up there. There's a port 23, no, no. It's still there. We're still using FTP. But yeah, it's interesting because there are those, you know, certain things you can do there today. And then, you know, AI is, again, great at making things, do things, doing things fast and learning things. But, you know, like you're saying, have somebody a human there to verify and approve stuff. And, you know, keep those guard rails to only do what we know is okay for it to do.

36:44And coming up with those guard rails is critical. You know, like critical thinking is back, baby. Actually, I never went away. But now you enabled by these other automation tools, whether they're AI enabled or or track automation, you still need to be the orchestrator and the architect of the process. You still need to be able to think through, you know, how do these things string together? So really set of critical thinking skills that are still super important for anybody in an optional, that opt-sec option and other. How would you boil all this down? Like, so take this home for us. What does it mean for net opt teams? What, what sense of urgency should they have? What should they be thinking about and the, you know, as the wrap on this conversation? So I'm going to go back to the, most of the network engineers I speak to say, I've been complaining for years that we need a better way.

37:44And I think right now, AI is making that more apparent to business owners and budget owners and IT leadership that now let's go look at a better way to do it. So I think the, the negative side of AI is actually driving people to look at automation where especially those that have not done anything for years and said, no, I know there was a better way why am I stuck with this? In fact, they probably, you lost your best engineers because they were busy doing that instead of, you know, actually building some of the strategic things that they came in to drive their careers and their careers are now just patching, right? So I think the pace at which it's moving, the pace at which these vulnerabilities are being released, the pace at which they're being compromised has changed the game where I think that's what it means. So I would say for network leaders that don't have network engineer screaming out loud, maybe you're not listening to them. And I think the question becomes,

38:45why don't we start looking for better technologies enabled by the same AI that's part of the problem to come up with more pragmatic solutions. So I do think that vendors that, I mean, companies that don't start doing something now will feel the pain even more so in the next 18 months. Yeah. 18 months might be generous. I would say maybe even 12 months. I think we're seeing a pace of change that's faster than, I've seen in my career. I won't pick her with 12 versus 18 months though. The time is now to really be thinking about and evaluating solutions based on operational outcomes. Or if you've rightly, you've both brought that out really clearly. And where AI tooling is the right solution by all means, don't box it out, find the right tools for the right job here for sure. So. Yeah. One thing I would add as well is,

39:45I think also teams need to work closer together. We've kind of siloed security, infrastructure, operations, finance, business, and kind of like, hey, I've got my thing, I've got to be my thing. Honestly, we've got to work together. Don't need to all come together and sing kumbaya, but at least we need to work together to solve this stuff. Attackers are going to be attacking the business itself, not just infrastructure, not just the security, not just the application. And so ultimately, we've got to work together to defend against this stuff. And part of it is being able to automate remediation. Part of it is being able to actually do remediation, not just take the security report, throw in the garbage, but talk through it with each other and help each other be successful, right? A lot of organizations want to spend on security, don't want to spend on infrastructure. Well, you need to maintain your infrastructure. Some organizations have AI budget, go do stuff with AI. Well, that's not just buying more cloud tokens.

40:45It's are there solutions with AI, with ODI, build automation, you know, answer these questions for your business to figure out what, how you can achieve those outcomes together as an organization. And I would say, you know, the hopeful Scott Robon here, I'm pretty much an optimist. I try to temper it with realism. Hey, we've got new tools coming to market that help us look at things more holistically, where I can be looking at my network flow information, my SIM information. I don't have to treat it in silos as much as I did before. And I cannot just find the needle in the haystack. I might be able to find the needle in the needle stack across IT silos that we've traditionally separated because of our own, you know, human limits, well, least Scott's human limits. Maybe you two don't have those limits. But like I'm hopeful that LLMs and specifically train models that incorporate network telemetry,

41:47security info, application performance, you know, there is this unified field theory someday that we're getting tooling. It's going to help us have one more holistic perspectives on problems and that will not repeat, will not exclude the security angle. So it's been an incredible pleasure to have you both here today. I know that I'll see you in Tucson, Arizona at AutoCon 6 in November. Do you have any other events that you're going to be at where people can seek you out? Yeah, so I'll hopefully be there at AutoCon with a couple of our team members. We're generally at the Cisco Live events and the checkpoint events. So if you've got a Cisco or a checkpoint event near you, we'll probably be there. If not, pay the world as virtual now. I could see you on screen here. And we can have a chat anytime. We should. For sure. Raka, anywhere interesting the year showing up in the near future? Yeah, now these are these are great places where we think network engineers are congregating more asking for,

42:49you know, more solutions from their vendors. And we certainly like to partner with Cisco and checkpoint on these pragmatic solutions. We got in them. Both of them supported us as we've kind of you know, supported their customers. Very good. Well, Raka and our fun. Thanks so much for being on Total Network Operations today to learn more about how Backbox is helping teams reduce operational risk and strengthen cyber resilience through automation. Go to Backbox.com and take a look. Keep an eye on them. They'll have more to share about what's next for AI powered network operations in the coming months. I'm looking forward to learning more about this. Thank you so much. Thank you. And to everyone who joined us, all you operators, thanks for coming and being in part of another Total Network Operations episode, networking needs networkers. I hope you've learned something from today's conversation. I certainly did. Please give us your feedback. You can contact me via a DM on LinkedIn or you can send us official followup at packetpushers.net slash followup.

43:49Thanks again and we'll see you next time on Total Network Operations.