TNO074: Lean Teams, Big Demands: Managing K-12 Complexity with AIOps (Sponsored)

0:00Welcome to Total Network Operations. It's the podcast for all you pack and delivery personnel. Our mission is simple to bring out great ideas and network operations. I'm your friendly neighborhood podcast host Scott Robon. And today I want to welcome Ben Shapiro. Ben is a senior network engineer at the Eugene, Oregon School District 4J, the sixth largest public school district in Oregon. He's also a misfit with HP networking. There are proud sponsor today. Thank you, HP. And he's recognized as a missed expert as part of their technical advocacy through leadership and community. So Ben, welcome to the show. Why don't you give us a brief introduction of yourself and how you have a really interesting path before you got into networking. Tell us a little bit about that first. Well, I'm so excited to be here, Scott. Believe it or not, I listened to a bunch of Pack of Pusher podcasts and even this one. So I don't

1:01know how I got here, but I guess we're here to have a conversation. I haven't been a network engineer for that long. I will be starting my sixth year as a network engineer in December. And I found it like all network engineers do through a variety of life choices. So do you want to start with the origin? Where did I start? Your superhero origin story is perfectly appropriate. Well, when I went to college, I was really interested in science and I was told I had to choose one and I chose chemistry. And I just did not love the eight hour long, like freshman chemistry lab on a Friday. And I started looking around for other majors in the sciences and I found geology. And so that's why I last shot you because it was like, hey, I got to do all the science. And I took that all the way up into a doctoral program which brought me to you, Jane, at the University of Oregon, where I'm luttled microbes on rocks with supercomputers. And I did that for

2:05like 90% of the doctoral program and then decided that I didn't want to be an academia. Roger. So I did the thing that was the opposite of grad school, which was make wine. If it's still used all the knowledge and modeling, I'd gained the fermentation process. And I did that for four years and that was glorious. But it was not really a great job from a salary standpoint. Let's just say for an art form and a blending of science and art, wonderful. I want to get back to it and I miss it every day. But I started looking around for what's my next adventure. And I was in a bar in Eugene, Oregon one evening. And I uttered the words free BSD because for the past decade, like in grad school and making wine, I'd been tinkering with networking and servers and my garage and someone in there perk there's of oh, Unix.

3:05And it just so happened to be the CEO of a fiber to the home startup in Eugene. And a couple of years later, it's like, I need you. And I was like, what? And I started working and I started two weeks before the COVID lockdown in Oregon. And I suddenly had to reengineer the entire network. And the first device I touched was an MX-480. Okay. And I did BGP Emissionary with maybe like what's BGP? And I was directed towards the RFC for BGP by the CEO. And 1771. Very good. Yeah. Yeah. Not against. It's great. That's right. And I realized that the RFCs are like an academic journal and I just ate them up. And that's where I didn't all my networking knowledge is just reading the RFCs like an academic peer reviewed journal. That's a great connection. I'm never thought of it that way. But you're right. And they often

4:07represent research and what but not how. Implementation. The bar of C's can vary widely versus what's in the academic research. Then there's the vendor implementation. Just get me talking about EVP and VX land and you'll find out. Oh yeah. That's how I got into networking. And I helped build a fiber to the home network in rural Lane County. Just the county about Eugene sits in. And I got 15% of the CARES Act money delegated for Oregon to build a lot of fiber out there. So not only did I learn networking, I learned how to design outside plant. I learned how to design negative 48 volt DC power systems. We don't have enough time on this podcast now given the more background you've just revealed to me. Maybe we should fast forward just for sake of recording to okay, that's great. You've got to know facilities and infrastructure. Somehow this guy you into K through 12 networking and IT.

5:11What was that connection? Yeah. So why I mentioned the power systems is I out engineered myself. I was often waiting for the outside plant engineers and they didn't want to touch DC. And I wasn't doing network anymore. So there was an opportunity. There's a vacancy at the school district and I had heard through the wire that they needed a lot of help. And there's nothing I love more than building and fixing networks. So that's how I got there. And it's really fun because they're like resounding theme of that somewhat circuitous path to becoming a senior network engineer. Has a unifying theme, which is studying and operating systems that you can't directly observe. That's geology. We can't observe the core of the earth. And why making like there many pitfalls? You don't know what the microbes are doing in there. You can infer what they're doing from observations. It's the same with network against packets. You can never possibly know exactly what's going on

6:14on a network at all times. You have to have proxies. You have to know what to look for. Well, so that led you to your current very challenging environment. K through 12 technology support networking. What pushed you there? You have to even zoom down and made all of these observations. You said, let it go to the most challenging IT ecosystem that I could possibly imagine. I'm sure that's what you were going after. Yeah. I had had this great building ISP from scratch, kind of a gain of knowledge, but I'd never operated an enterprise network. Let alone had to get in and fix one. It's really curiosity is what brought me over there. We're some of the shiniest objects that look interesting that I've never had to touch in fiber build out or waiting for the power people. EVPNVXLAN.

7:22We don't usually go straight there in K through 12 networking. What the heck are you talking about? Yeah. EVPNVXLAN is an interesting beast. It gives you all of the redundancy you get if you're like a service provider, but still lets you have a platform platform network. It's very complex. There's a lot that needs to align for it to work properly. Vendor gets back to the RFC conversation. We had just a little bit ago of vendor application of the EVPNVXLAN is a very different between vendors and there are deviations between the standards. That's the stuff I love. The issue I walked into first was, decoded to me as the multi-cast storm issue. I knew it quickly wasn't a multi-cast storm and then it became a treasure hunt of what's going on?

8:28What do they have? This is in a K through 12 environment? It was. It no longer exists. It's a TLDR. It wasn't the right tool for this organization. It is fairly complex. You wouldn't expect academic school district networks outside of a data center to need something like that. You also found that this wasn't the only set of challenges in the K through 12 environment. He'll let that on the in a way for us a little bit. Yeah. That gets down to the meat of like we had this complex thing. It wasn't right and started a kind of adventure of reimagining the challenges that we had in finding resilience versus complexity and having that spectrum to play with. We're dealing with buildings that were built in the 1950s.

9:33In many cases, before networking existed, our data frames are custodial closets that have a sink. Sometimes they're storage. They're very unsecured. They're usually open frame racks that don't fit modern switches now. We don't have a lot of structure cabling and some of the older schools. So just access ports are an issue for growing needs. There's a lot of cap-side E around. We now have demands for multi-gig capabilities on structured wiring. We've got wiring that's out of spec. That's beyond 100 beaters. That doesn't have the special fancy cables. That can do that. Yeah. Everything's Wi-Fi. Wi-Fi, the location of access points. Many of our sites were designed back when we were doing 802.11b. We were only using 2.4 dickers. Now we have much shorter wavelengths that don't travel as far and they hate materials.

10:40That's right. Yeah. The tenderation helps us back from challenges. Just for linking up building infrastructure. One of the things so insult added to injury here, you also have more and more applications and services moving to the Wi-Fi and IP network, which you totally educated me on that when we spoke initially. What are all these other services that actually have to run reliably on this aging couple together infrastructure? Yeah. I mentioned COVID changing the ISP network that I had worked on. COVID had the same effect on K-123. We onboarded a lot of digital tools that then followed us back to the classroom. There were no infrastructure upgrades to accommodate that. So there's a lot of technology dependence on instruction now. That's one vein. The other vein is

11:42building automation, paging intercom, then analog systems that existed for 34 years. In these schools, like originally units in some ways. Pretty reliable. They've been dying. They've been reaching their end of life. There was no plan to replace them. A lot of these replacements have been kind of like fine debonning a crash deploy it. The only systems you can buy now are IP and multicast based for those. We have lighting controllers. We have kilns that need to connect to an AWS server somewhere and they talk IOT chat in plain text. We have a Thomas Flores scrubbing robots. We have 3D printers. There's all sorts of IOT stuff. We've Phillips Hue bulbs because they wanted having ambiance in the school. The harshness of the light. This kind of makes you king of IOT,

12:48which I wouldn't have even thought of that. You've got so many different light types. I forgot the HVAC. You can't like configure an HVAC now without connecting to the cloud first. We found this when we built the three schools on a local bond levy. One of the schools, they had finished the HVAC. They needed to test like the airflow or something with it. They figured out they needed it to connect to the cloud. But they hadn't pulled fiber into the building. They hadn't finished the power to the data frames. There was no terminated structured cabling. You can't configure this until you can reach the management system. You must have security issues buried in here too where certain device types shouldn't be reachable by other device types. How much care and effort do you put into segmentation, microsegmentation, or do you not have the luxury to take the scalpel

13:51to the different application groups? This is a really interesting subject because it kind of weighs on what your fiscal resources are. Right now, we're facing a really difficult two years. One of the other effects of COVID, there was a lot of one time stimulus funds for education. That's kind of like in the bucket of capital expenditures. Those were used for op-ex in many operational expenditures. There was no sustainability plan. Because of that injection of cash, we on-boarded costs that we couldn't sustain. Now we're seeing the deficit as a result of that. We don't have a lot of monetary resources to secure a network the way that a four-shed 500 company would. You have to start playing a game of simplicity, complexity, and resilience and security.

14:58Really, the things that I care most about are in our data centers. That's where we're investing in that security plane. Then we have to think of more creative approaches out in the sites. Because we have 38 sites that are physically spread out on dark fiber in our metro area. We would need 38 firewalls if we really segmented it the way we should. Sure. You have a cost constraint that everybody runs into budget issues except maybe the financial ops space. But the screws are probably a little tighter or maybe a lot more tighter for anybody in Tate control except for probably the most affluent communities. Even that's not a foregone conclusion for sure. We made it this far without talking about AI so we can pet ourselves

15:59on the back for that so far. But I am interested in, okay, from an AI ops perspective, what's really turning into providing utility for people in your position? Are there tools, things that are coming around from an AI ops perspective that are helping you with all this? We can talk about things that are helping and things that are not helping and not there yet. You can tackle that anyway you want. What do you think so far? Yeah, so I think this is a great bridge for the topic of monetary resources because I think the first challenge I face is that I'm really the only full stack network engineer. What I mean by full stack is like BGP routers down to wireless and across all the different OSI layers. And we have 16,000 students. We have about 2,000 staff that's probably less now given our

17:04rebudged in and you know, lame off of people. But it's an insane ratio that a lot of organizations like don't have. And you face a point where the two options are AI ops or more bodies. And I know the more bodies solutions not going to happen. So I did do a really good job of selling the story of why AI ops is important. And you'll see in a lot of marketing, they'll have like, oh, reduced tickets by like 400%. That's not how I view the value AI in that sense. AI does the TDS cross comparison of weird stuff on my network that I would never have time to even know existed. It's like finding a needle in a haystack that you never knew was there in a way. Didn't even see the haystack let alone know that I needed to find a needle in it. So can you pull out any like, give us a couple of concrete examples of a weird thing that you

18:11actually found AI ops tooling really helpful with. Yeah. So the AI ops that I have flaxed on to is a product that is very mature. And the place where we have the most churn, where we have the most exposure, which is a layer two campus and branch switching. That's it. So we chose Juniper Mest of that time. Now it's HP Juniper Mest networking. I don't know what the string awards is at this point. Mist seems to have survived. So I think you could still say Mist. Yeah. Yeah. But yeah, it's been a great tool because it has this ability for you to onboard your current Juniper switches to just send telemetry for the AI ops. You don't have to do the full templatized automation management stuff part of it. And when we started adding our,

19:14kind of, disparately managed Juniper access switches to Mist, we had to slow down our pace of onboarding because we started finding marvelous actions that were being triggered. One of the most prominent was the marvelous action called bad cable. Okay. Uh oh. And the crazy thing is like the efficacy of that marvelous action is so great. We've never not found a bad cable. There's always been a bad cable somewhere. That action won't tell you where it gives you that like starting point to see in your brain of why was this triggered. And then our infrastructure specialist will go and be like, Oh, it's because the keystone jack got like shoved into the one gang box or like someone rolled over a patch cable with a chair. There's always something off. And it's never been wrong. And when we started onboarding those switches, we got like 16 bad cables. We didn't know existed. And

20:15it would just keep piling on. We're like, whoa, we have to slow this down and address the success that come in. So the marvelous action there just to make sure I understand it. Is it, is it being overwhelmed with notifications? Is it shutting down a port when it finds a bad cable? What, what is the action actually do when a bad cable is detected? Uh, it's just a notification. That's the center of your organization. Um, you know, I either do like old fashioned email blogs so you can do a webhook that like pushes into your ticketing system. Uh, which is pretty cool. But it's just an alert like, Hey, I found a weird thing. Um, right. And yeah, it, it's really great. And kind of, I think that summarizes like where I view AI right now, like AI always gives you a starting point. And I kind of tell, um, like technology staff who are interacting with the, the, the ms dashboard that when you see an action, that's your opportunity to pause and ask, why did this

21:17get triggered? And that's your starting point. I, I think that's a great encapsulation of where we are with AI tooling in, in that ops right now. It still keeps the human, you know, in control. Um, it's gone and use read only information to say, Hey, Ben, this is something you really need to pay attention to or even not that, you know, not poke, poke, poke, just, Hey, Ben, yeah, you're something interesting. If you want to go investigate it. Um, and you've already proven that you're, um, you're very inquisitive in nature. So if you, if it's pointed out to you, you probably will go check it out. So yeah, no, it's, um, it's weird. Like just yesterday, I got a, Marvis action alert. That was, there was an SSID like injection, noticed by one of the APs that, uh, with them one of the schools and I had like garbled characters and like percentage signs. And I was like, Oh, this is weird. Should I care about it? And I used that same tooling to finally

22:17come to the conclusion is it was seen once it didn't reoccur. And it was probably a malformed or corrupted frame with a valid, uh, FCS part of the package envelope. So it's not always right. And you always have to just, uh, use it as a starting point. It's a place to start looking. But the important takeaway is like you would never have been looking in the first place. And it helps me triage what's important. Are there other relatively common actions or notifications that you're seeing? Like those are those are some, I can see those happening. Your role, your chair rolled over a cable example is probably all too common in your environment. I would guess. Oh yeah. What else, what else hits you in your team? Maybe not as regularly, but, you know, things that you weren't even thinking about before that you're getting notified for. Yeah. So something that I struggle with that I think many watching will also agree with is

23:22interacting with vendors. I think, um, there are a lot of vendors out there who got their start when bringing a device online was plugging a cable in. And now that analog devices networked and there's not a lot of great expert teeth about how layer two networking works. And having a little buddy that's trained on five billion data points, hold a vendor accountable is priceless. We saw this in, uh, the rollout of these replacement paging intercon systems that use layer two multicast. And, uh, there was a site that got installed and I got in a work from Mara saying, Hey, there's a multicast fly. Like look at this change in traffic and I'll actually show you a graph. I mean, like I have, I observed this change at this point in time. And, uh, kind of coincident to that, I got an email from that vendor saying that I didn't have any multicast

24:24configured on my switches. Um, clearly, I did like, well, well, the demist did it, you know, and they did not have their multicast settings configured correctly. My conversation went something like this, Hey, I noticed like there's multicast flooding on that switch. Please check your, you know, settings because there's something that's not configured correctly. And they came back like, Oh, no, all our settings are configured properly. And then my response was, I'm going to trust you or my little buddy trained on five billion data points. You tell me the little buddy wins here. Maybe just maybe. Yeah. Well, so what does the path forward look like for you? You know, we've, we've talked a long time about, you know, what is self-driving actually look like? And maybe that's become more popular with actually cars that are full self-driving today. Um, but in networking, you know, I'm former junior burly, we've been talking about a long time. And, you know, I, I feel like, you know, if we,

25:28if we get academic, you know, from a calculus perspective, we're going to, we're approaching a limit, right? And full self-driving is definitely a limit. And you can walk halfway there. And then walk halfway there again. And then walk halfway there again. What are, what's going to help you build trust in letting your little buddy actually do things in the network that change production versus just letting you be notified and digging deeper and taking action as humans? What does that look like for you and your team? Yeah, that's a great question, Scott. Um, I know that, uh, people love using AI for everything these days. I don't. I, I, I see it as a tool. Um, and, uh, to draw a good analogy of how I think about AI helping me in networking is how I use it to, like, help out with server administration. Uh, there's nothing more human, in readable than log files.

26:30And you can feed a log file into an LMM and to like, it will make it readable for you. It will find the two lines and 10,000 lines of log files that matter. And, right, I think that frames how I wanted to help me out with networking. And so there are some tedious things in networking that just waste time like bouncing ports. Yeah, the auto negotiation things things on the layer too well is like, oh, I have to click a button and bounce a port. I want to do that for me. Like, that sees like data inconsistencies or if it sees like weird PLE power draw or tears negotiation mismatch, I wanted to do something for me. So I don't have to one notice something weird. And then to like drill into the switch and the dashboard and then have to click a button. That's where I would like it to start to do all those tedious things that take away

27:30from the time that I need to kind of make sure we're ready for the next demand our organization has. Sure. Sure. Sure. I share that vision in a in a multi domain context as well. Like, you know, we used to have to just look at what's happening over the Wi-Fi segment or just what's happening with security logs or just looking at application performance info. And I think the tools that we're talking about here are going to help us in our helping us span those boundaries to not just find the needle in the haystack, but the needle in the needle stack. You know, when the needle looks like all the other needles, we have to find the one or the couple that, you know, at the 10,000 logs or the 100,000 logs, but a million or 10 million logs, right? I'm very, very hopeful and encouraged about things I'm seeing there, analyzing piles of data that we all, you know, human cognitive limits

28:33keep separately. You know, nope, that's a geology problem. Nope, that's a winemaking problem. But this is a networking problem. You said earlier, you were able to zoom out and see connections across all those domains. I'm hopeful that we'll keep making progress here on that front in particular, not just that, but that's like, that's one area that gets me jazzed here, for sure. Yeah, Tilly. It's why I think you need to be a geologist to be a good network diagnostic technician is to be able to be both in like a detailed view and out at the 30,000 foot level and be able to keep your foot in both areas. The part of AI that I'm most excited for is when I heard Bob Friday announced that HPE networking was working on a large experience model, not a no on that, not a large language model, a large experience model. That's when my like, little ears picked up and

29:33I would like that multi-domain collection of data to be able to look at my network, especially staring at the like the later two Wi-Fi access portion of it. And we like, this is what your network can support today and then be able to feed into it like, hey, we want to do this with our network. Will it support that and have it say yes or no? And then back it up with like, oh, you need to upgrade your platform of switches or you need to add two switches to this closet or you need to upgrade the bandwidth or lower the latency. On this pathway, I think that would be fantastic for a planning standpoint and also for like, sure, keeping some accountability for decision makers and like, actually, we can't do that and it's not me saying no. It's like billions of data points saying that's not a good idea. Right. Have you had to play that card in conversations with your leadership? I've my spiky sense tells me that maybe maybe you have since that's top of mind

30:38for you. Like that seems like they gives you great confidence, right? To be able to say, oh, it's just an event's opinion. It's we've got we've got model or models that have been trained across multiple installs. Has it been helpful to you in real leadership conversations? Yeah, I think anything anytime I have data to back up an argument, it's easier to defer that liability that I just don't know what I'm talking about. When you're interacting with an organization who's a mission, it's a very different than technology. There's a lot of suspicion about, is this true or are you bending the facts? And if you have data to back it, it's so much better. And I know that HP Network has been working on the ms dashboard to have like human, rightable SQL. So you can even dip more into them this database to pull it out. Like I trialled it

31:45when it was in development. And I was able to pull out data from this that could tell a staff member, your Wi-Fi experience is going downhill because all the students in this high school are congregating by your office at this time, which is when your Wi-Fi stops working. And then I was able to take that. They were related. We fixed this by putting an access point over here, but we'll need X and structured cabling and X and hardware purchasing and Z and OpX cost. And I was able to give that to the decision makers. And ultimately, they're like, it's not worth the money to us. I was like, okay, I didn't say that. And having data to back up your claims, that's just good hygiene. Right, this is not an AI-specific conversation. Good reminder of not just Scott's opinion, not just Ben's opinion. Yeah, but like the amount of work I would have to do to home grow a system,

32:50to collect that data, like would never happen. Right, right. Especially when you talk about these different types of models. Experience model, other types of world models, or even smaller, more focused LLMs. There's lots of interesting possibilities there for sure. You do have a lot of interaction with the MIST team at HPE. I know you're part of the MISTFITS program. Can you talk about that a little bit? And tease out two things. What that looks like with your interaction with HPE and getting stuff into product, much more quickly or effectively. And also the shoulders you get to rug with other MISTFITS and bouncing ideas. I'd love to hear both of those elements of the program. Yeah, so I am a MISTFIT. I joined later than some of the like original MISTFITS who were not wrangled into a sanctioned group. I joined kind of when that adventure started with

33:54Genre Per. And I'm surprised it made it out of the pilot program because we are boots on the ground, customer advocates, I guess. And we are very opinionated and we are not shy about sharing our opinions. And that's been really embraced by HPE. They look to us to provide feedback to the PLM's on their presentations sometimes. Like there's all sorts of ways they integrate us into building the MISTFITS product line. And it's a whole bartering system and it just works them out. I don't know how it does, but it's an amazing thing and I haven't experienced anything like it anywhere else in the industry. And you are taking on risk as a vendor, even making yourself a little vulnerable with your user community. Because when you ask for honest feedback, guess what? You'll get it by from people whose jobs rely on, this has got to work in my now.

35:01Yeah. And it's just been it's been a wonderful relationship because there's no kind of hierarchy to how we can engage people in the organization as kind of like whoever then wants to contact, you can contact. And I think that keeps the whole process of product development honest. Because you can kind of attack it from both the top and the bottom. For sure. I've seen features that I've requested like show up in production three months later. And it's great. Like one of them was like, hey, I need a historic graph of DOM power data from my optical transiverse because I operate dark fiber between switches. And there's just something that the PLNs didn't think about. They thought like I was in a closet using deck cables. Yeah. You have a unique infrastructure to get between buildings. So you need that

36:04information that is probably not in most other IT environments. So yeah. And what about other misfits? Like if you ever, do you do you get together and plot? I mean collaborate on hey, we would really like to see this in the platform. Or hey, that's a great idea. I would love to take that and try that in my shop. Any any contact points like that. Yeah. God. It's the best community. I say they're like my people. And the thing that frustrates I think most of us is we don't get together enough. We're too spread out across the world because it really isn't an international group of people. And it's a really small community. Like we started off with only 25 of us identified. And it's only grown a little bit more. I think we're still under 40. Don't quote me on that. But yeah, it's a really tight knit community within us. Like we have our own like communication channel that is HP sanctioned that I think we probably forget that everyone at HP can see it

37:11sometimes. But that's okay. But yeah, we're constantly asking each other like, hey, I saw this weird thing like have you seen this? And then we'll have people from HP engaging in that conversation. So we're both finding emergent weird things that are on the platform or talking about changes or talking about ideas for feature requests. And just like how we're using the platform. And just shit, it's a sharing of ideas all around. And it's the best. They're my people. I wish I lived to white next door to them. That's awesome. I mean, I totally, I'd say four or five years ago, I when I was on the vendor side and really in an SE and SE leadership roles, I don't think I understood the power of community like this. And now that I'm out as an independent hearing about programs like this, plus seeing things I'm involved with network automation forum, USNUA,

38:14YCO and SharkFest. Like the ability to get people together just to bounce ideas off each other, whether it's about a vendor product or a system or the industry in general. So super useful. So yeah, I'm sold, you know, and I'm a little jealous. You know, you found your people. That's awesome. Yeah. Now I, I, I, whether I'm typing at the networking community is to begin with, I know you've been to Nanog. Like I have gone to Nanog when I've had the ability. One of the downsides of being in the public sector is I don't get to many conferences. I should go to, but it was just a really great community in general as an industry. And that's great to find people. The misfits have a monthly meeting. And that's where we'll do what you talked about. We'll we'll we'll gang up on an idea of it. And like, Hey, yeah, what Brian said, I experienced that too. And it's a good way to get like, quorum behind an idea. For sure. For sure.

39:18Well, if I take this back to, you know, okay, what are what do you have to do on Monday morning, you know, or you know, now that school is getting started and you've got, you know, so many plates spinning to reference Ed Sullivan show reference to probably nobody gets anymore. You know, you've used the term democratizing networking. How does what we've talked about help you do that? And what do you mean by democratizing networking within your team? Yeah. So I like to say that HP Juniper missed democratizes the network for me and spend my like broken record as a part of the group. It's like my marketing statement for it. Right. Because I think a lot of the issues I deal with on a daily basis are educational. Wi-Fi is magic. The network just works. You know, we don't have to plan for any infrastructure updates because it just works. So it gets to the cloud. You know, and a major part of that as educational gap, at least an art technology

40:22apartment is transparency. So miss to gives everyone in our department visibility into how the campus and branch network is doing. Anyone can sign in and there's a chatbot that can help them make sense of what's in there. And like my dream is that before escalating a ticket to me, anyone in the technology apartments asks, marvelous, like, why is this user having a problem? And then if they can't make heads or tails of it, they paste the response into a ticket they open. Because I don't have to be the only one watching the network. We have to find ways to be more creative. And it also means that my other backend compatriots can also help with that. Section two, instead of knowing how to interact with the Juno CLI and knowing set commands and knowing how to check them and how to roll back configurations, right. They can just know what port

41:24profiles go with like what type of ask and they can just click on a little port on the dashboard and head safe. And then they've closed the ticket that would not be a good use of my time. Sure. That seems like a really good and practical process step to enforce. Hey, before you raise this to me, ask marvelous, please, please just try. Like if you, if you had a, would you put a percentage, how often does that really happen amongst your team? There are some frontline tech sports who have really embraced it. There's a lot of work to do in that right. Because like the challenge is like everyone's overextended, like it's another step of something to do. Like we just want it to work. But how I've been trying to bridge and educate people in technology is when they do escalate a ticket to me. I start with the missed SLEs for wireless or for the wired switching

42:25and then take a screenshot and then everything in my description is in the screenshot. And that's been really successful. So they might not be asking me more of this. But they are looking at the SLEs and insights tabs within my dashboard that give you a lot of actual action of old data. And sure. That's what really got me with that platform is I did a pilot. And just to see like what data would get me. And I couldn't believe that all the data was usable because I'd come from a controller based Wi-Fi vendor that had a data telemetry collection thing that I was running on a bare metal server that had like seven hundred eighty six gigs of RAM and 64 logical cores. And it gave me zero data on 2000 access points. Zero-actional data. Like I my the way I troubleshoot Wi-Fi

43:25in that system was just throwing my hands up the way I don't know. Now what not what you're looking for for sure. So so just having good data being able to have people who are not very technology adjacent field to interpret the data like given the SLEs. And then being able to ask a chatbot for further explanation goes a long way to to marketizing a lot of the network. Sure does. And it's it's it's not going to change anything on production. It's not going to go rogue. It seems like that's a good habit to get people to into in the context you know along with your people who are already feeling overwhelmed right try to get them there graciously but you know here's what can do for all of us not just you. Yeah and the best part about having missed a manage switches is that your manager can install them. Sure. So you mean on board that.

44:35Same with the Wi-Fi access points. It's that's what's great is like I don't need to be present for a switch deployment. My role now is to write the templates and then have documentation that explains like what the template does and like what template you apply. And then literally anyone can be a switch or Wi-Fi access point installer. It's if any of the you know teacher staff need extra hours right they can they can do it to just kidding I'm not trying to task them with more work. So that's a hidden part of the democratizing the network. The labor democratizing the labor. Yeah. Now before we started doing that like during a medium-sized school would take like a week because I would have to like console into every switch and like hand right. It's a configuration and then check it and then I would like make some human errors that would make a dot work and

45:35pound my head against the wall for like an hour. And now I don't have to do that. Well why don't we wind this up by going back to the beginning per se. You're at a time of recording you're at the beginning of a new school year. What's coming around the corner for you and let's think about in terms of this school year and then from a planning perspective what do you want to get in place for next school year. What are the next 12 to 18 months look like for you. Yeah that's a good question Scott. It's again I've said it it's a hard time because of budgetary constraints but it's also a hard time to even replace any infrastructure because we've been struggling with the funding mechanism that the FCC runs for schools to buy technology at discount prices. It's called e-rate and it's run by an arm of the government called the use hack and we did our e-rate

46:41application back in like February of this year and you're supposed to be able to receive equipment at the beginning of summer. We still haven't gotten approval for our application so we're struggling to replace our 10 plus year old juniper switches with misdynate of switches and replace wireless with mist wireless. So that is I'm going to be planning another e-rate cycle before I even have the last e-rate cycle approved. So I'm going to be spending a lot of time doing paperwork and calling people but on a more like functional side I'm starting to feel like the part of my network that isn't mismanaged that would be the like distribution portion the security appliance like edge firewalling portion the bgp border router portion. I want useful insights that mirror what I can get in mess so I've been spending a lot of time with telegraph and flex

47:46db graphana trying to find ways to also get actionable data in front of the rest of the technology department. So other people can keep an eye on the network and I don't have to be that sole responsible party. Well look I really appreciate the view that you brought here you know both with the tech and the operational constraints because they're not you can't disaggregate them. We talk about disaggregated networking another time but you know your constraints are your constraints. You've really educated me on the whole package here so I appreciate that before we finally close you know any final words of wisdom for anybody might be in a similar environment give them hope things to watch out for drop a drop a truth bomb. What do you want to wrap up with? The hardest thing that you can do is make something simple and flexible but it is totally worth it.

48:53You don't have to have the shiniest thing you have to have the thing that's right for your environment and it does not have to be complex and it does not have to be inflexible it can be both those things but it takes a lot of work and thinking and research to get there. Right that down we don't have to write it down we just recorded it. Definitely definitely words of wisdom. Thanks for your time Ben today are you do you ever blogged are there places where you'd point people to you for other good info? I have aspirations but sadly I'm really overextended at my day job and I struggle with that currently that's where I'm trying to work on myself. Hopefully one day I can impart some of this knowledge in a place where others can use it but that doesn't exist right now. You did a little bit of it here and let me thank you personally. I really appreciate you can always find me on LinkedIn and you know we're going to take conversations out of Ben from

49:56there. Excellent. Well thank you Ben so much for being on Total Network Operations today. Thank you for all you folks who tuned in listening and watching on another conversation here on Total Network Operations. We'd love your feedback. You can hit Ben or me on LinkedIn. For packet pushers follow up please feel free to hit packuppushers.net slash follow up. Thanks again and we'll see you next time on Total Network Operations.