Why the network industry is struggling to fully adopt network automation

I got to attend Autocon zero recently, which if you're not familiar is a new event by the network automation forum, and what an awesome event it was. I mean, How much better can it get than hundreds of network nerds getting together to talk shop? Now the actual event was about network automation as the name implies and specifically about why the industry wasn't farther along with it. But there were actually a couple themes that emerged as well. So with me today is Peter Spurgata, head of product at INTential who noticed some of those same themes popping up in the main sessions, but also in those side conversations too. Now, I can't promise that we're not gonna talk about AI, so get ready. My name is Philip Dhruvasse, and this is telemetry now.

So Peter, thanks for joining today. It's really great to have you on, and it was great to meet you for the first time a few days ago. And like I said in the intro, talk shop a little bit about what we were seeing at AutoCon. So thanks very much for joining.

Get Hey, Philip. Thanks for having me. I'm, I'm super excited to be here and, you know, anytime I can sit around and just talk tech is a good day.

Yeah. It really is. It really is. I mean, I try to do that at home, and then I start to glaze over and, people start to exit the room, conspicuously, doesn't doesn't really work well.

So then I I have to resort to the online forums and social media and stuff, which is like a love hate relationship these days anyway. So yeah. You know what I mean? So you gotta tell me before we get moving because when I heard your name, but didn't see the spelling, I'm like, okay.

Peter Spargata, that must be Italian. But then I saw the spelling. I'm like, that's not Italian. So what is your name heritage?

It's amazing how many people do think it's Italian. And, you know, I can always ask the question. Have you ever heard of Spregatta pizza? No.

Course, but, now the the spelling of my name has changed over the years, and and it was originally spelled s z p r y Uh-huh. Uh-huh. And and so, you know, I tell people that, like, oh, yeah. That's definitely Polish.

So I am a Polish heritage.

Okay. Great. Great. So, I mean, the premise of the event maybe I'm talking at a turn here, but from what I understand, the premise of the event was why aren't we farther along with network automation today?

The networking industry specifically. Right? What's funny is I was talking to one of the organizers, Scott Robon, and I said, hey, Scott. One of the things that you guys should do in, like, sessions or something.

He's talk about why we aren't farther along in network automation. And he looked at me, he's like, that's kinda what this whole thing is about. So he gently corrected me. But, I mean, I sort of agree with that.

Like, thinking about it, and, oh, I mean, hence why I brought it up to him. Like, why aren't we farther along? And that's one of the things that I wanted to talk to you about because, well, frankly, I mean, your company, attention, is eyeballed deep in automation and programmability. In fact, that's why Kentic, my company is partnering with you to kind of the convergence of observability, programmability into that closed loop workflow that network engineers all dream about.

Right? So we can spark an event, see what's going on, understand it, actually push some sort of configuration, validate it, and then get back to keeping the lights on. So why do you think We're not farther along with network automation today.

Yeah. I tell you, you're asking the million dollar question. There is no doubt about that. And even the sessions that we saw it at, Autocon was definitely one of the undertones of, I think, every presentation no matter what the presentation was on, is why aren't we further?

And I can actually trace my network automation routes all the way back to twenty twelve. I've been doing this now for twelve, thirteen years. I mean, that's just focusing on network automation, and it is frustrating at times to look back and wonder why we aren't further. But to answer the question, it really comes down to a couple of things.

First and foremost, I think it comes down to fear. Fear of what automation really means. It's new. It's different.

It's scary. It's fast. It's aggressive.

So, you know, it's in in networking professionals, I think, generally speaking, are conservative by nature. You know, without question.

Mhmm.

The other one, and and this one's a little bit more tongue and cheek to a degree. But I think there is some basis in it is that, you know, many individuals that got into the networking industry, you know, started out in school and they took a CS, you know, one on one class, their first programming class, and realized hell no. I don't wanna do this for the rest of my life. Yeah.

Right. Right. And, they move into networking and low and behold, here we are. X number of years later, and and now we're saying, okay.

Yeah. You gotta become a program. Like, wait a second time out. What's going on here?

Or Yeah.

I think there's a couple of misconceptions out there too. So I think the fear I mean, some of it's legitimate. Right? But some of it's unfounded. Like, this idea of like, oh, fail fast, fell off and be hipster network engineers and, like, you know, the chaos monkey thing, well, that's not necessarily true.

Mhmm.

But if you say that to, like, a network manager or IT manager whatever at, like, a major hospital.

They're like, no. We can tolerate exactly zero risk. We don't fail fast fail often in a competitive facility. Right?

But that's not the idea that fail fast fell often. It's not literally go around breaking everything. And then, oh, let's see how we can improve. That's the idea.

The idea is to be more proactive with trying things that's in the context of rollback plans and in the of carefully planned projects and things like that. And I think that might be kinda mixed into this incorrect understanding of what network automation is You know? So, I mean, you know, you can automate the collection of information from a closet of switches. Okay.

Why should there be fear there? There's no real impact.

And then on top of that, it also kinda misses the point that the network is so impactful to application delivery to the actual end user that if I just do a Notion on the wrong interface. There goes my job. Right? Mhmm.

Yep. So you don't need network automation to have that fear. I did enough cutovers where I had plenty of fear being an engineer without automation having to do it. You know what I mean?

You know, it's it's funny you mentioned that because, you know, as someone who has done no on the wrong interface. Oh, I know. I I appreciate that very much. So, you know, there's there's a couple of very interesting subtle points there.

And, you know, I think one of them is that, you know, you you talk about this idea of, you know, automation and and process and kind of the two things going hand in hand. And one of the fear factors, I think, we're starting to realize with a lot of teams is the fear stems from and you you kinda touched on it. Really stems from the fact that they don't have the right operational practices and culture and policies. And and and those things in place and automation starts to expose that.

For so long in the networking industry, we've been able to overcome that by sheer will. Right? The the CLI command line, you know, interface hero who can jump on the box and hammer out the commands at at lightning speed. You know, has really masked a lot of the problems that we've got with the way that we approach, you know, managing network infrastructure today.

Yeah. Yeah. There there actually was quite a bit of discussion about culture. In hindsight, I should have expected, I guess, because that does pop up, you know, like the dev ops definition is largely about culture.

And, I heard one person talking about cams. Right? So that there is that component that network automation is incorrectly viewed as well. I got a couple of scripts.

I need to learn Python. And, yeah, there are scripts in Python involved possibly Python with network automation, but the idea is bigger than that. It's the workflows and processes.

But I don't wanna go too far and then just kind of equate network automation and orchestration. One of the main speakers there called it out. There was somebody from the New York Times, and I actually became friendly with him, and I and his name escapes me at the moment, but he spoke, and it was great. And he sort of made the distinction between, you know, a collection of scripts and pushing out scripts, which is basically like the CLI at scale. So it wasn't that compelling and then orchestration.

So you, Peter. How would you personally define the difference?

To put it, you know, in in the simplest definition that I can because, I mean, we could probably write paragraphs, if not pages, if not books on this topic right here. But I think to put it in its most simplest form, you know, automation is really all about automating task. It's all about task management. Right?

It's all about automating tasks. And orchestration is all about the coordination of all of those different automation functions to achieve a business outcome. And and so that's how I would phrase it. But I would back up one moment, and and and I do want to challenge one thing here.

And that is we tend to take this stance that scripts is not automation, and and I disagree with that to a degree. Scripts is still automation. One of the things that prevents the industry from moving forward is we don't have an end state and goal in mind. It's okay if All automation means to you is writing a ten line script that puts a description on an interface.

Maybe that's all you need, and that's okay. Yeah. It just kinda feeds in that fear factor that if I wanna do automation, I have to be a software engineer, and I have to have version control, and I have to have CIDR pipelines Those are all good things, but automation is really a mindset in terms of how you want to operationalize your infrastructure. I never wanna lose sight of that focus.

Yeah. That makes sense. In speaking to, I guess, what you would call, like, medium size enterprise engineers, and I I use that term medium size the way Cisco does. Because, like, they, in their documentation or in the textbooks over the years, I read, you know, I'm studying for a certification, and they'll call, like, a medium sized business, and then they'll equate it to, like, a ten thousand an organization. I don't know where they get these.

You know, it's crazy. And, like, meetings are, okay, whatever, with a million dollar IT budget. But So when I say medium size, I'm talking about, you know, some organizations with several thousand people. So in my world, that's still there can be some serious complexity and sophistication there.

But not an unlimited budget, like a web scale company or a huge global service provider. So what you have other than the fear So maybe there isn't fear, but maybe there's just limited resources. Now you have engineers that are like, I, you know, I've been doing this for eighteen years, and and my goodness. I don't have the time to learn this stuff, and I don't have the resources to go implement this because I am busy keeping the lights on, which ironically, a gentleman there on the panel, Carl Newell made the comment.

You don't need to keep the lights on if they're automated. I quoting him. He said something a little bit differently, but that was the spirit of what he said. It was a great quote, though.

It really was.

Yeah. But ultimately, that that is a problem as well. And speaking to some enterprise engineers, like one gentleman who's a, a senior network, manager or engineer at a health care facility here in the northeast, It was just like I love this stuff. There's no way I'm able to just start tomorrow or when I get back to work because I got a hundred other things to do. So I think that's a problem too. The reality of limited resources in those smaller and then, quote, unquote, you can't see my air quotes medium sized organizations.

Yeah. I mean, it goes to that premise that automation, while it's all about doing repetitive tasks or Linux software do repetitive tasks at take away valuable time from network engineers from doing actual engineering work. You know, network automation, even automation in general needs to become a state of mind. Right? It it needs to become a way of thinking about how we do things, and that's what starts to drive some of those behavioral changes, those cultural changes that are necessary. To expand on that point, there was also another presentation that I thought was really interesting.

And and I forget who was doing it, but he had put a slide up on the screen that really resonated with me. And he had three boxes, and he talked about, you know, engineering layer, and he talked about process, and he talked about management. And the point he was making is that all of those pieces have to care. Right? It can't just be one engineer off in the corridor driving an automation strategy really needs to be something that is embraced by the organization from the top down, the separation of automation orchestration, understanding their place in the world, and then start to layer on additional capabilities, like observability, if you will, on top of those things, to start to create that closed loop where we start to achieve some of those those outcomes we're looking for.

Yeah. Absolutely. And, what would you say are some of those outcomes then? Like, so I I'm gonna I'm gonna prepress my question saying, I I don't necessarily believe and agree with this idea that the purpose of network automation is to, like, automate away the mundane tasks so I can do the ones that I'm more interested I don't really buy that because I've been an enterprise network engineer for most of my career, and I'm going to work.

You know, I I like what I do. I loved being an engineer. It's not that. I wasn't breaking rocks in the in the field under the sun, you know, with a hammer on my ankle.

But, certainly, it was still a job. I put a meme out there recently where Like, network automation helps me to automate away all the mundane work, so I have more time to do other mundane work.

I'm still at work. So I don't buy that as the real main compelling reason that a network engineer should adopt network automation from a technical perspective and then as you said, from a cultural mindset. So what do you say is the goal or are the goals of network automation?

If there's one thing we can all agree on, no matter where you're coming from in the networking spaces, the stuff is getting more complex by the day. Right? And we're a long, long ways away from the three tier architecture way back in the day of, you know, Core Ag and Edge, and we could keep the entire mental map of the network in our head and and knew everything that needed to be done at any given time. So when we start to think about all of the different domains that are coming into play now and all of the things we have to do, and and it's not done the same way anymore.

Right? The way I manage my SD WAN infrastructure is not the way I'm managing my cloud network infrastructure. It's not the same way. I'm still managing some of my traditional network infrastructure.

So I think this is where network automation really can start to play out and offer huge value to an organization because it allows you to offload some of that knowledge into your platforms and let your platforms take on that responsibility, that you can continue to focus on what I think you got into the industry to do, and that is to engineer networks.

Yeah. But what you just said, correct me if I'm wrong, is that I can use a network automation workflow and whatever that means from a technical perspective to unify the operations of divergent systems. I mean, that sounds like a really heavy lift. I have one vendor a SD WAN vendor b data center network vendor c on my campus. I'm doing a rip and replace of my wireless, so I got a vendor d now in the mix just on wireless. That sounds like a really heavy lift. How does an engineer even get started and wrap their mind around that?

It's a huge lift. No question. I think you just hit at the crux of the separation of automation and orchestration. Right?

And and this is a lot of what we talked about at at Autocon is I can automate my domain and I should automate my domain where it makes sense for me to automate my domain. It's understanding that there is a layer above that, a horizontal layer across the organization that allows me to orchestrate across those domains. And actually now when I get to there, and it takes time let's be honest. This is not something that's happening in a in a day a week a month, you know, in six months.

This is a committed direction for an organization. But when I get there, we start to see, I think, for the first time, real attachment of business value to what's happening at the network layer. Right? And I think that is such a key statement to make because for so long, we struggled to understand what's happening at the network layer to the business and add value to the business as opposed to just simply being a drain and a cost center.

Yeah. That makes sense. And I would say add value to the business, sure, and then you get your approval from your management folks. But for the folks that are turning wrenches, both virtual and physical, right, they need to see some value as well. And they're not gonna necessarily be as concerned about the business value, but there can concerned about what am I getting for all of this time and effort that I'm putting into it? Oh, I can push config faster.

Okay. Or I can get more information into my IPAM faster automatically. Okay. That's cool. But I got three or four engineers working on my team.

We we can do that pretty quick. I have to spend three weeks learning this new tooling in order to do that a little quicker. Like, who cares? I think that's actually one of the reasons that there isn't as much adoption is that a lot of engineers don't see as much value in just you know, pushing some config unless it's at, you know, huge scale.

I think that the value is gonna start to be seen more readily and clearly when we start to see these systems whether they be in, you know, one huge system or or the partnerships like Kentic and INTential or, even huge homegrown systems where you start to have this direction toward an automated root cause analysis. Maybe one day, the pie in the sky for engineers has always been automated remediation who knows if we'll ever get there. But where we're actually looking at the state of the network whatever that means, that's an entire series of podcasts unto itself, taking direction from what we see from events and alerts, things like that, and then making adjustments to configuration, actual devices, virtual physical, ephemeral into configuration, and then validating that everything's working properly.

The way I see it is the end of the day, this thing that we love that we call the network, which is completely distributed and all wacky today, is ultimately just the substrate. It's the mechanism by which we deliver applications to human beings or computer to computer, but nevertheless, it's data application That's really what it is. So I wanna know what's the state of my network, what's the issue with that application delivery, or whatever it happens to be, push some config automatically, and then tell me if that works. If the application's performing the way it should, that's very, very compelling to me.

And that's where I wanna see things go.

I think it is. And and there's no doubt that you can start to attach real value here, but here's where I see the challenge. And and I would love to kind of peel this one back a little bit is that, you know, when I sit down with network engineers, whether they're they're starting their career or they're thirty years into their career, it is surprising to me, and I'm over generalizing. So let me start let me make that statement.

But but it's surprising to me how many network engineers don't take their heads far enough out of the bits of the network to understand the systems that are running on top of them so they understand how to add some of that value. It's like, I can build the network. I can troubleshoot the network. I can optimize the network.

I can do a lot of that, but I don't really know what's going on at layers above me and I don't care. And I think that that's one of the flawed lines of thinking that many network engineers have. They really need to start to understand, you know, what are the needs and requirements of the lines of businesses they're providing, you know, infrastructure for. Yeah.

I mean, that's what network observability does specifically is put all of the telemetry that we derive from the network, network adjacent services and devices and components, and it's very, very diverse and divergent and voluminous.

Right? And we put that into the context of application delivery. First and foremost, there's other context. We look at cost and stuff. But ultimately, it comes down to what is the state of this system and how is it impacting application delivery and ultimately an end user's experience.

And then when there's a problem, you know, let's push some config. Now another issue is that I, over the years, have heard folks but mostly vendors and mostly on the blogs and podcast say, well, I don't trust the system though. So, you know, I just want a red a big red button that says let me review, send to my cab board and then push the config. Sure.

You can push it automatically. It'll run all these, like, cool playbooks and stuff. Awesome. You know, I want the final say.

And the funny thing is that the more actual, like, in the trenches operators I speak to, especially as what, you know, Cisco calls the medium size and smaller enterprises, I don't think that's really an issue. Like, I've spoken to folks that work at large organizations, universities, huge school districts here in the Northeast, north near New York City. So very big. Health care.

And they're like, wait a minute. So I got a problem with my my wireless and whatever it happens to be, and you're telling me that the system will throw an alert, let me know and automatically push config. And then fix it. Yeah.

Take my money. Right? Like, I get it. If there's a gigantic, like, you know, I've had all sorts of complex overlays in my data center.

I I need to be involved. I get it. You know, there is an element of risk aversion in in certain scenarios, but I don't think that operators are struggling as much with trusting the system as I think some would suggest.

Yeah. I I would agree. I I, you know, I it's not necessarily about trusting the system. I think it's least my experience. It's more about trusting the external factors that they can't control.

Oh, yeah. That's true.

You know, that's that's really where I think a lot of those that those trust issues come into play. I also think it's, you know, it's it's very easy in hindsight and in in retrospectives to be able to look back and say, Oh, we should have been able to either see that coming. We should have seen that outage coming. We should have seen we should have been able to easily automate this thing or that thing from happening.

And and I think we forget sometimes that it's a lot harder to anticipate some of these things than it is to look at them in hindsight and make decisions about, you know, how we could have done better. That's one of the I think one of the real big challenges that networking teams have because they're constantly in that reactive mode, and they're not proactively thinking about how do I make my environment, you know, more bulletproof? How do I make it self remediable? If that's a word.

Right? It's one thing to self remediate, but it's another thing to build an architecture or design or an infrastructure that is self remediable.

Yeah.

That lends itself to the ability to remediate itself if and when you finally apply those types of workloads and stuff. But I I do agree with the bulletproof thing that you said.

Both from experience and from years working with colleagues who have that opinion that the primary goal for a lot of engineers, maybe some of our listeners disagree. I don't know. But In my experience, the primary goal for most network engineers is a reliable, stable, predictable network.

Very rarely do I hear performance as the top goal? They're like, yeah, whatever. I got a crap ton of bandwidth. We're good.

You know? We're yeah. Of course. If there's a latency problem, we're we'll fix it. But as they say latency is the new outage, reliability is the key in stability.

That's one of the things that I've heard year after year after year. And then from my own experience, again, as the primary goal. And I think that once folks, engineers, start to adopt the culture and mindset of network automation, and then, of course, a subsequent technical components as well, they're gonna find that that's actually gonna help them reach that goal of reliability and stability. It's not a joke about pushing config, which is cool unto itself.

Oh, look how much I was able to change all these prefixes, my BGB prefixes, right, and whatever. For all these devices at once, and that's cool, and there's a fulfillment in being able to do something efficiently.

But ultimately wrapped around all of that is a more reliable overall infrastructure. And and I think we gotta remember that when we say network and network infrastructure, we're talking about a huge variety of devices different vendors, some of which we kinda like light lightly manage, you know, with some cloud construct that we have limited access to. Maybe it's just campus boxes that are still physical. I have console cables and a console server.

Who knows? So there's an incredible amount of complexity not to mention the services that are important to the network like DNS and load balancers and maybe your iPad and things like that. So there's a lot of stuff going on there that can lend itself to unreliability that can relend itself to instability. So I think the adoption of this culture mindset and the technical components of network automation is gonna help us get us there.

To a more reliable and stable network. And and that's even in networks that aren't hundreds of thousands of end users around the world, just in your, you know, your ten thousand person organization.

Your medium sized enterprises.

Yeah. And I say that and I use the word just with air quotes because the reality is I I was a power engineer for many years. There's a lot of complexity in those multiple data centers. Oh, for sure.

Mission critical wireless. The largest hospital in my area that owns a bunch how they all consolidate. They have a bunch of hospitals. They have, like, eighteen thousand end users, and that's the largest medical system in my area.

That's eighteen thousand. Cisco, that's an SMB, but but that's yeah. But they're like operating rooms and heart transplants. I don't know if they do heart transplants, but you know what?

Is helicopters landing on roofs. This is all mission critical serious stuff here.

It is. It is absolutely. And and, you know, it's one of the things that disappointed me coming out of of some of the sessions at at Autocon. And and not just there from customers as well is this idea that here we are in almost twenty twenty we're at the end of twenty twenty three.

And my goodness, we're still talking about how to get to closed loop automation in in in the fact that we still haven't gotten to the point of understanding that, you know, it's one thing and and I think you touched on it. Right? It's one thing to configure a network. And and you're right.

There is a lot of satisfaction seeing the BGP peer come up. Yay and seeing my routes exchange. Alright. I'm excited, but it can't end there.

And it is ending there, and that's disappointing. Right? It it really needs to take that next step because we have to be able to have a sense of what the infrastructure is doing. Otherwise, there's no way we're ever gonna get to some of promise lands that, you know, we all want to get to, whatever that may be.

Whether that's self driving networks, I think it was discussed at Autocon or self healing networks or or whatever, you know, self, etcetera, you wanna put there. In in addition, it also starts to lend itself to understanding once we can make this stuff almost second nature, it allows us to start looking at what is the next thing we can layer on top of the network infrastructure. We're not there yet, but I'm hopeful this is the beginnings of us being able to start to move in that direction.

Oh, yeah. Yeah. We're definitely moving that direction. I agree. Even if it is just in terms of awareness, Right?

When we saw it ourselves, there were almost four hundred people at Autocon, and many of them because there were several, like, questions where they said, raise your hand if you're doing this or raise your hand if you're doing that and and the surprising reality is that a lot of people aren't doing that much stuff, but they were there. They were there. They spent, like, real money to be there. Whether their company spent it or they spent it.

They spent money to be there and move forward on their own network automation journey, whether it was from zero, the starting line, or progress from wherever they were. So I'm encouraged. And the fact is that I I saw, you know, there were a lot of vendors there, of course, not a lot, but there were a handful of vendors there that sponsoring the event made it happen. And so going around those tables and talking to folks like you guys had potential and some other companies and really understanding how there's like a whole community that's built around this.

There's an entire ecosystem of stuff online and and people. What a small world it is too.

But I think the resources are there to move forward as well.

But one of the things that I heard in a couple presentations, you and I chatted about it, little bit different from network automation and that what we've been talking about is the introduction of AI artificial intelligence, maybe more specifically LMS and how it can or maybe is fundamentally changing IT operations.

Now that is a can of worms that I just opened. They're all over the place now.

So let's try to pick them up.

I was just thinking so we're gonna go there.

Well, it came up and that is the, the big thing to talk about these days. And you know, the thing is I work in in marketing now. Right? My title is director of technical evangelism, but I have been a network engineer for many, many years. I've been on the side of the table where marketing or sales came in and did a little song and dance, and, you know, I wasn't interested. Wanted to know the hard facts and details and show me the peak you know, let let me see how this thing works and solves my problem. And so when I heard things back in the day about SDN and some other different technologies over the course of the years, a lot of it was smoke and mirrors and eye rolls, but I am seeing actual technology with regard to LMS and AI being integrated with various systems, not even necessarily networking, but, you know, tech related and tech adjacent systems and actually have a positive impact And so I'm wondering if this is sure that's on the Gartner hype cycle.

You know, we can we can track it there. The trough of disillusionment. Maybe that's where we are, but I actually think that there's something here very much so. I mean, what what is your opinion? Do you think this is smoke in mirrors, or is there some value to AI and LLM specifically in the realm of IT operations?

Yeah. You know, it's it's a fascinating time to be in this industry. There is no question about it. And and when you step back and just look at it, if I take off, you know, my engineer had my product and and all the different hats that I wear.

And I just look at it as a fan of technology. It's exciting to see what's going on. I think that there is applicability to it. The question, you know, that immediately comes up in my mind, though, is, you know, and if I think about the state of networking today and, in fact, you even touched on it in a previous comment about trust issues.

Right? Is the network, and and are we at a point where we're even ready to have this discussion? There is no question there of thinkers out there, and and there are some interesting applications or applicability of LLMs to networking on the periphery. But Is it gonna really start to make its way into mainstream?

I'm really struggling with that at the moment. I'm not saying it's not. But I'm fearful that we're getting out way over the tips of our skis now if I could use the old analogy there, with this.

You know, I'm gonna disagree with you, Peter, on this one, because I'm thinking in terms of networking. Okay? So maybe not as applicable in other areas, but I'm a network person, so this is where my brain goes. I think number one, if we're talking about it, it's not too early to talk about it because we're talking about it.

That's a circular argument. But we're talking about it. Yeah. It's a technology. It's a technology.

It's a real technology. We can talk about natural language I mean, engrams and stochastic reasoning and grammatical logical reasons. These things are real and exist, and they exist in math. We can look at the algorithms.

Now are they relevant? Or are we looking over what is it? Over the tips of our skis? What did you say?

Yeah.

I went over the tips of our skis. Yeah.

By the way, that was the first time I ever heard that.

And I'm gonna Oh, really.

Yeah. Yeah. But I I disagree that I think it will soon become very impactful to IT operations and is already beginning to. I'm thinking specifically with the incredible volume and diversity of data that we ingest, from network devices, network adjacent devices and those services that we rely on. And, of course, like visibility, observability companies. Right? The amount of information and the types of information and the variety of formats coming into our, underlying UDR is tremendous.

Mhmm. So how does a human being engineer figure that out and interface with that data? And I think So let's take AI out of it, but just applying, like, data analysis workflows and some of the statistical analysis algorithms that we learned as sophomores in college. So nothing fancier than that.

Still going to be a great aid in moving forward and understanding how these data points relate to each other. So for example, I have millions of packets per second coming in Over here, I have a flow data. Right? And so I get that.

And over here, I have some, like, metric from my DNS server. So these are all very different. So we normalize them and trans that's all machine learning pre processing. I can look at seasonality.

All that really cool stuff, but that's still difficult. And I think LLMs, especially as they get more sophisticated within the domain of of IT and networking, right, because that's gonna take time to reduce hallucinations and increase accuracy. We can interface with that data more quickly. And something I've said on many podcasts before is if I had a team of thirty engineers that all had PhDs from MIT, I could probably get away with that and they can do the work and figure out all these answers.

But even then, it would be slower than a machine could do it. So I think as, like, anything else, it's iterative I do believe that as we experiment with using this technology in networking and in my case observability and then, of course, fine tune and and adjust I use that term fine tune specifically because that is something that we do with LLMs. Right? We fine tune the, the temperature, for example, so we can reduce hallucinations and other things.

That I think there's gonna be significant value for an engineer just saying, what the heck is going on? What is this data telling me? Mhmm. And then using natural language to do it.

I don't know how soon that's gonna be, but Peter, I say this all the time. I don't know if you're a star trek person like me, but I love how commander Jordie Leforged, lieutenant commander, depending on which episode.

He talks to the enterprise computer, and he's like computer. What's the deal with the UPS couldn't do whatever he's in. And the computer does its thing and gives him an answer. And I'm like, it doesn't mean that he isn't an engineer anymore. He still has to solve the problem, but think that's one of the ways that we're gonna see some improvements in IT operations.

I could be wrong, but It would be great if we did.

I think, you know, one of the things, though, that leads me down this path of skepticism is, you know, when you think about and and I'm I'm talking about networking specifically now within the broader domain of IT, But when you think about networking, there is still a lot of networking in terms of the design, the realization of the design, the implementation, that is still a lot more art than it is signed. And that's where I start to become concerned about, you know, sure. You you give me a long enough runway.

I'm sure we can train LLMs to be able to understand, you know, the the numbers of of permutations in someone's head for, you know, making sure that this app continues to run or that app continues to run. The question is is is it worth the effort to get there? I don't know yet. And and that's a lot of the thoughts that that I'm having in terms of this is gonna start to realize itself in the infrastructure.

You listen to some of the big thinkers in our industry and around AI and ML. And, you know, we got to hear from one of them, you know, at at Autocon Creedy Capella. And I think that you listen to the way they present it and you walk away thinking it makes perfect sense. But then you get back to your desk and you sit down and say, how do I apply this to my world today?

And that gap is so big still. Yeah. That's true. You can become lost in this this journey to the point where you turn yourself in circles enough times that you end up doing nothing.

That's where I become worried about, you know, how is this gonna move forward?

Yeah. That's a good point. I do have some friends. I'm not gonna call them colleagues so they don't work in networking, but I have some friends, one in particular, PhD from I think Texas, a a and m, Texas, whatever.

He's from Texas. But, he works for a I don't wanna say the company's name, but it's a large global company with over three hundred thousand people, multiple BUs. One of their BUs is oil and gas. Another one is nuclear health care, things like that.

And they use ML, which is the technical component of the broader term AI to find correlation and predict medical issues, specifically looking at what are they called? MRI scans and data based on who you are your age location, race, things like that. And they use that to find strong and weak correlation, causal relationship. And it's literally a mathematical correlation coefficient.

So there's a use case and it's It's very, very valuable to them. So I think the technology is actually already there. It's how do we integrate this into what we're doing in IT. We have ephemeral information containers that live for a short time or maybe the IP address lives short time.

And here's the thing that I agree with you, the whole art form thing. There is a subjective component here, like the end user experience. Is this web application a little bit too slow? Or is this web page?

Or the example I've used in the past is should I care about this alert? Should the system send this alert to an engineer? So if I have two one hundred gig links in a data center and they're an active standby, which I think is poor design, you wanna use them both. Let's say you have a a two hundred gig links, and you have one of them being utilized as your active and your standby is dribbling along at literally one meg per day as an average, and then it spikes to ten megs.

That's a statistically huge increase, but absolutely irrelevant to application performance in the end user. Well, there's a subjective component there. An engineer would say, I don't care. Don't worry about How do you add subjectivity into literal math that lives in Python in Jupiter notebooks?

And how do you do that? So I think the whole form thing is a challenge. But we're we're solving that too. I mean, what maybe we don't alert on it.

But maybe if we see it go from ten to twenty and then twenty to thirty day by day, we see a trend, and then we alert. You know, I don't know. I don't know the answer. But I I do agree with you.

There's a lot more complexity.

The cool factor, it makes sense, How do I actually apply it and get value? That's a little harder. Yeah. Yeah. Especially for an engineer that's sitting there literally trying to keep these lights on. And when I say lights, I mean, a little green Hopefully, all green lights on their switches. Right?

That's absolutely right. Absolutely. Right. If engineers are struggling today to adopt automation because they don't wanna write thirty lines of Python. Getting them to AIMO is gonna be a challenge.

Yeah. But I don't see them adopting it. Like, right now, they're gonna go learn Python gonna go take a course at, you know, Coursera or, you know, learn Python on the hardware, whatever. You can go learn Python.

Maybe start with terraform because that's easy. Whatever it happens to be. So those are certainly there's effort, but it's not like you're going out and writing algorithms and applying ML models, and you're not doing that. I have a feeling that that's gonna be as a service in the sense that organizations, vendors, networking vendors, observability vendors, automation vendors are gonna integrate that within their own systems and offer it as part of their, you know, their overall platform offering.

That, I do agree. Yeah. That, I do absolutely I I don't foresee organizations. At least, again, I'm overgeneralizing, but a majority of organizations embarking on a ML strategy that is unique to them. It's gonna come from products. It's gonna come from services. It's gonna come from their vendors.

Yeah. Which is tough because you gotta train a model on your own network.

Think it was Greg Ferrow who always likes to say that every network is a snowflake.

Right?

Yep.

In the sense that eighty percent of all the networks are the same. It's TCPIP switches routers, hubs, hubs. Did I just hubs. Wow.

You're not the one now. Well, you're always here. I don't know. We're going back in time.

Hopefully not hubs in your network, but eighty percent of all the networks are the same, but then you you always have that twenty percent, like, where I have, you know, the pets versus cattle thing, you know. And, oh, I remember that switch when I installed it. And, you know, I got some really interesting PVR and some custom route maps that I lovingly, you know, came up with.

And so when you have that twenty percent, that's different network to network. Yeah. That implies that sure. You have models that you can apply in the general, but then for your own specific network, you need to train that LLM to be able to answer questions relevant to your network.

And then underneath that LLM, because remember, the LLM is just the NLP section. That's just the natural language interface. Yep. Need something that's able to actually look at the data and find the correlation and do the fancy cool stuff.

So, I mean, what do I go out and buy a product and then it'll have it sit there and train it from my own network for a month or many months. I don't know. I don't know. There's a lot of questions there.

A lot of questions.

There is. There there absolutely is. Is is skeptical as I am. Again, there's that technologist in me, though. It is still exciting to see and think through what the possibilities are. And I think that's maybe the most important takeaway at least for me around AI and and what it means, you know, long term for the networking industry is that we can start to see some of the possibilities.

And as we continue to work to get our houses in order around people processing tools, we can start to, you know, potentially floor, you know, what, some of these utopian visions could be for how we build and and operate infrastructure.

Or dystopian depending on how the AI thing goes. Indeed.

Right?

Yeah. No. But, you know, I I think it made sense that that conversation was there at Autocon, though, in a conference or event all about network automation, they they kinda go hand in hand. I mean, AI is, you know, this concept of a closed loop workflow with auto remediation and intelligence within the system.

Beyond just an intelligent control plane, you know, rerouting my traffic, but looking at correlation and making a decision based on rule sets and all these things, It makes sense that it's in a conversation around network automation. That is a workflow of looking at events and then pushing config to adjust events. We're just adding that intelligence. Over the top at some point.

So, yeah, I I think it made a lot of sense, and I really enjoyed talking about it. I got corrected in the AutoCon slack because I made a comment about how you can remediate or mitigate hallucinations.

Yeah. There was somebody there much smarter than I who corrected me and showed me, some some resources that I could read So a really cool learning event as well for me, in that regard. So I hope for you as well, by the way.

It was absolutely, you know, getting an opportunity to really hear firsthand what some organizations are doing. And it's something that it's nice to finally see us getting to this point in our industry for so long Yeah. I've been doing this for the better part of thirty years. And, you know, I can remember a a time when you didn't talk about what you did in your network.

That was your secret sauce. You know, you didn't talk about your design or or how you built your configs because that was intellectual property. You've never let that information go. And it's nice to see that and large, I think we've overcome that at this point.

You know, people are starting to realize the benefit and the value of community and sharing and and idea exchange and network automation forum really did a good job bringing these groups together and giving us a nice wide variety of sessions from, like you said, everything from AIML to, you know, observability to understanding automation and orchestration. And and the whole event really came together in in a fantastic way.

Yeah. Yeah. I agree. And, and then, you know, based on the vendors that were there, there are options to work with a partner to adopt a network automation mindset, call an actual technical practice using the help of a vendor like Kiteential, that literally specializes that and has the team behind it to make it happen, the professional services, I assume.

Right? And, therefore, the skill set to make that work for your particular network it it was a lot of fun, and I'm looking forward to Autocon one. I assume that they're gonna call it Autocon one since we started at zero. I don't know.

But assuming so. Whatever they call it, I am looking forward to it and and looking forward to being there. It was cool to get to the event and immediately saw these are my people. Felt it was very familiar, and I and I enjoyed that.

So anyway, Peter, it was really great to have you on today. I hope to speak to you again soon. But for now, how can folks reach you online if they vehemently disagree with something that you said a warning.

Please do and and and reach out and and and point out everything I'm wrong. I love to hear when I'm right, but we all grow by hearing one more wrong. That is for sure. But, yeah, you can reach out to me.

Probably the easiest way is on Twitter. I'm at private IP. So you can hit me on Twitter. You can also hit me on mastodon at private IP, and that's the easiest way to get in touch with me.

Great. Thanks. And you can find me online. On Twitter at network underscore fill. You can search me on LinkedIn.

I am on Blue Sky, Ios now as well, and my blog network fill dot com. If you have an idea or an episode of telemetry now, or you'd like to be a guest, I'd love to talk to you. You can reach out at telemetry now at kentic dot com. So for now, thanks for listening today.

Talk to you soon. Bye bye.