Network automation was a really popular topic in the networking industry maybe nine or ten years ago, but then it sort of fell off the radar for a while. Now I don't think that's because it became unpopular or we all, as an industry, decided against network automation or something like that. I think what happened is that we got to work on the courses and developing the training materials, the mechanisms, the new workflows to actually automate production networks rather than just talk about it in blogs and podcasts, and the networking vendors responded in kind. Today, though, it seems to be coming back as a popular conversation piece of networking.
But what's cool to me is that rather than theoretical conversations, we're seeing and hearing success stories, discussions of workflows and mechanisms and tools and skills that work or don't work, and basically how engineers are automating their networks in the real world. So with me today is Steve Leeper, a friend and someone I worked with years ago at a VAR here in the northeast US, an experienced network engineer. He's a speaker at several USNUA tech meetups, and today, the lead network engineer implementing network automation for a large enterprise.
So in this episode, we'll talk about why and how to get started with network automation, what works, what doesn't work, what tools you need, and what skills and mindset you should develop. If you're interested in getting started with network automation, this should be a great episode for you. My name is Philip Gervasi, and this is Telemetry Now.
So Steve, thank you so much for joining today. It's been a back and forth trying to get you on, because you've been really busy, which is really great to hear. I mean, that's why I wanted you on is because because you are eyeball deep in this stuff. And, and I have to say that I also miss a lot of the stuff that you're telling me you're working on. I remember working on and I really miss that. So thank you for joining.
Before we get rolling here, can you give the audience a little bit of background about who you are, what you're doing right now?
Sure. My name is Steve Leeper. I work for a health care, payer company in the, Northeast United States.
You know, I kinda got started in IT like a lot of people do, you know, building computers in high school. That was interesting, but oddly enough, didn't go to school for it. I was a broadcast journalism major, worked in television news photography, for several years, won some awards, and finally decided I wanted to change careers. Always had an interest in IT, got into kind of the typical, help desk kinda roles that people start in.
Got a lot of interesting, experience doing that type of stuff, progressed into doing some desktop support before ultimately landing into networking. I always found networking really interesting for some reason and had just leaned towards it and knew that's where I wanted to go to in networking.
You know, worked at the college that I, went to school at, doing networking there for a while before moving into healthcare.
Worked at a large, hospital system in the, Upstate New York area, which is, definitely interesting experience if you've never worked for a twenty four seven operation where people's lives literally are on the balance of your wireless network. It is an interesting experience, but, I do not miss the, three AM, upgrade windows. But, it is a interesting experience. I think a lot of people should try, if they have the opportunity.
From there, my girlfriend that time, now wife moved, which made me change jobs into a a VAR position, which is very interesting. Another type of, aspect to IT that I had not dealt with before that is very interesting to see and really get you into a lot of different network areas. Got me deep into wireless, which I had done a lot with, and then, ultimately into my current position, also kind of back in healthcare again, but more on the, insurance and the payer side.
You know, and the the thing is that I remember, having a I mean, I had a similar experience to you in that I started, a completely as as far as an adult in my professional career, I started in a completely different field as a high school English teacher. In my mid twenties, switched over to IT. And then, you know, eventually got, specific into networking. But I remember, before I did that, working a helpdesk job for a while. And I I really feel that that was just so important, so foundational for what I did after that because I got to touch so many things and get involved with so many aspects of technology prior to focusing in on networking.
How about you? I mean, was it the same for you?
Yeah. I've definitely had the the same experience. I between working at a help desk, doing a lot of application support and desktop support and especially a lot with Windows, you do get a lot of experience with those different types of areas, of things that run on the network and how they may interact with the network. You know, for example, we did a lot of stuff with wireless in the past.
Gives you know how to configure different client side settings on Windows, whereas, you know, other other network engineers I have have worked with don't have that experience. They're not really sure. There's, like, well, I only know the wire Wi Fi part. Like, I can help our desktop guys compare how to configure those types of things, you know.
My background in television, we have a lot of AV equipment in a lot of our multimedia, you know, Teams and Zoom connected rooms. You know, all that work from home and video conferencing stuff. I have a lot of experience with that. That helps a lot.
And even not necessarily expertise, but just knowing some of the terms that are out there, have an idea how things work, has been pretty helpful in just helping other people troubleshoot when the network, you know, always, of course, becomes the thing that people wanna point as the problem. Oh, absolutely. They say, well, what about this thing Yeah. Over here that I've seen before that's not the network?
Yeah. And the reality is that all of these systems really interoperate together. I mean, it it's it's kind of odd that we focus in on just the network or just the cloud or just applications and say that this is a world unto itself. And the reality is that these are all systems that interact together in one giant system to deliver that application to the end user.
So having having that understanding, even if it's at a help desk level, which I, you know, granted is not super ultra deep, I think is still so important for then eventually getting specific and specializing in one area. Do you remember, by the way this is for our audience's sake. I think you remember. I hope you do.
I was starting to get into Cisco ICE, and everybody was, like, upgrading from ACS and Prime and things like that. Right? And we were doing a lot of ICE projects. Oh, by the way, also for our audience's sake, Steve and I worked for the same VAR for a time, which is where I got to know him.
And, I I had never done it. And you know how it is when you're with when when you're with a VAR, it's trial by fire. It's like, hey. Can you install this thing?
And I'm like, no. I've never heard of it. And the project manager is like, well, yeah, but you're the only engineer that's available that week. So now you're the subject matter expert.
I'm like, okay. I guess we're gonna YouTube that and and go install it. But But I remember doing an ice project, and I don't remember the customer now. And it was the first or second time I had ever done it.
And Steve pretty much just walked me through that entire project. Do you do you happen to remember that?
I don't remember the specific project, but I do remember being kind of the, SME when it came to Cisco ISE.
Yeah.
And it's just one of those things, you know, you do it five or six, seven times. You run into the same kind of things, and it's just like it becomes easy after a while. Even though I remember the first time dealing with some of those problems, it just you wanna bang your head against it. Right?
Oh, absolutely. And then, you know, two two times doing a project like that, you are the official SME to all the project managers.
And the funny thing is in a complex project like that, whether it's ICE or some other, you know, component of networking, it doesn't take that many projects before you really do become intimately familiar with a lot of the caveats and and gotchas, if it's a if it's a larger complex installation.
No. It doesn't. And one of the things I always kind of enjoyed and also found stressful about the VAR world is the first time you worked on a a technology, I always try to, when I could, try to lab it at home or somewhere where I could play around with it. Mhmm. I always felt a little bad using my the first customer as a guinea pig because they expect you to be an expert. So I I always try to at least have some sort of hands on with something before I went in the door somewhere.
And that kind of has led me down an interesting path the rest of my career. I've always wanted to be able to try out and lab things. I think in networking, we always tend to use the the the the production network as our test environment. Yeah. Yeah.
Absolutely.
So so then what was the main impetus for you to to transition more to a focus on network automation, which is the topic of the podcast today and the reason I wanted to talk to you because you've really been getting into that and and deep into it.
You know, we we discussed what it was for you to move from from a completely different career into IT and then from kind of a generalist at the helpdesk into networking. And and in more recent days, though though I know you're focused on traditional networking as well, a big part of your role now is in network automation. What what was the reason that you that you started down that road?
So it's a bit of a a long story, so I'll try not to take too long for this. But I feel like I've been kinda interested in just IT automation, and scripting, I guess, for a long time. I mean, you know, at the helpdesk and even in in desktop support that we did, where I worked, originally at a at a state university in New York, we did a lot of customization of our Windows images. And as a result of that, we just script a lot of stuff to deployments to deploy those devices.
So I learned a lot from those guys about writing Windows batch scripts and command scripts to customize every little piece of the application that would get deployed. Right? You open up a Microsoft Word. Every setting is set the way we want it versus having the default setting set.
Right? So that kinda got ingrained in my mind. How you do things like that, that it's possible, out of the box isn't necessarily your only option.
Going into the helpdesk, you know, I actually did some administration of our desktop, and help desk management ticketing system and wrote all these sort of automated workflows to just make the whole process simpler, right, and scripting that. And that was an interesting experience. And then, another thing I hadn't mentioned in my background is in I was a volunteer firefighter and EMT for about eighteen years.
And anywhere I ever went, I was the IT guy there. Right. You know, one of the departments I was out in the, Central New York area, we had a pretty interesting record management system built on MySQL.
And there's all sorts of licensing involved and whatnot to get data out of it. And I was like, well, it's MySQL. I can write scripts in a web page and some stuff to just generate reports and have people be able to just go to a web page and look at it. And that ended up being pretty helpful.
Learned a lot about databases doing that, which, again, not thing I would have done with networking, but it's at again, database, good skill to have. In IT, they exist everywhere.
Right. And then as I moved to networking, just having these ideas of just things you can do with automation and data just kinda led I think it was a natural progression, right, of being able to to get into that. Yeah. And then my current role was really focused on integrating network devices and tooling into some of our other automated, workflows that we have.
Where I am currently on our sort of, I'd say, our Linux and cloud side, we are heavily, heavily automated. But networking really was a very manual process, and it was sort of slowing them down.
And so the idea was how do we take everything that they've we've done in all these other areas and bring the network into that so that we can, automate a lot of our processes to make them a lot quicker. They had taken the process of deploying a server, which used to take days to they could have sent up a server without anyone really touching anything in about an hour, except for, you know, obviously, if we need a VLAN somewhere, you need a new IP address, like, then the whole automated process stops. You gotta get that information from networking, and it starts again, which is kind of an interesting thing if you think about it too.
Like, all of I most areas of IT, I don't know if about all, automation has been there in some capacity for quite a long time. And networking you hear this discussion a lot. Right? Networking has always kind of lagged behind, and that's been a good seeing both sides of it, it just got me to think of, like, there's definitely gotta be a way to make this work.
Yeah. You know, and how do you how do you get to that point?
Yeah. It's interesting that, you come at you're coming to this, specifically, network automation, not for the sake of network automation, but from the more broad idea and understanding of how to automate processes and workflows in technology and IT and saying, well, I mean, this has nothing to do with networking per se. It's just making life easier for engineers running all this stuff.
So why do you think it is that networking specifically lags behind? I'm asking you that question. I already have an answer in my head, at least at least my own, theory, but I wanna hear yours, of course.
Yeah. I've heard I've heard a lot. The big thing always pops into my head just because I'm a down in the weeds technical kind of person is it's very easy to mess stuff up, and then you don't have connectivity to those devices anymore. Right.
A server might be a little bit easier to have some sort of easy out of band iLO card or something. If you break it, you can go back in that way. But networking, if you break your routes to a router at a site that is two hours from you, are you gonna automate that and risk that, or is it easier for you to go in line by line and you can, you know, have dinner with your kids that night? Right?
That's what, to me, has kinda been the big thing of, like, it's easy to break connectivity if you do it wrong. Whereas, I think, servers and other areas of IT are a little bit different. You don't lose your entire access to the platform it's running on.
Yeah. You you wiggle the wrong wire while you're walking through the hot aisle. Right? You when I say we're like, you literally brush up against the wrong wire and there's, like, a, you know, a fiber that wasn't fully plugged in.
Right? Your data center's offline. Right? It's very impactful. Networking is the entire, mechanism by which all these other application servers, services, databases, they're all running on top of, especially considering today, most of the applications that we use are are over a network.
I mean, very, very little lives on my computer in front of me.
And so I would agree with you that one of the reasons that networking has lagged behind in automation is because, well, there's this cultural aversion to wiggling wires and and touching the network that is just so impactful, specifically to operations, like, in real time.
Whereas a server, like you said, you know, it it's a VM that you can spin up again or, you know, you can, you know, vMotion from do we still use VMware? I'm sorry. This is how out of date I am. We're still we can move things around much more easily than we can with the network, especially if we have a network that's been neglected because it's so impactful to touch it and modify it and change it and make it more resilient.
So we just leave it alone. Follow nothing is down, so don't touch it. Leave well enough alone. So let's talk about, you know, you're working on some recent or rather recently, you've been working on some network automation projects, at the health care provider that you're at right now.
Can let's get into that and talk about, specifically, what was the impetus to start automating? And you already mentioned a couple of reasons, right, to make things easier. But what were some of the reasons that you went in that direction?
Was that the nature of the job when you first took it in the first place? And, and, you know, let's talk about those details.
Yeah. So the the position I'm currently in, I think one of the things they've been looking for was someone to kind of bridge the gap between that network team and the sort of the the Linux and cloud engineering teams that were already heavily automated. You know? And I kinda had a little bit of that background.
So that's where that kinda started. So a lot of it was just kind of figuring out, okay. What is our Linux engineering, cloud engineering teams doing? Where are the sort of roadblocks with the networking team, and how to how do we kind of bring those together.
I had not done a lot of actual, scripting and programming network specific beforehand. I was familiar with a lot of stuff. So the first thing was how do I get up to speed? Do I do Ansible? Do I do Python?
Some other framework or language out there.
And the thing that worked for me the best was to find out what are we already using and come to find out that those were some of the tools that were already being used in the environment. So Yes. Right. I already had a good base of people in the Linux and engine cloud engineering teams to kinda go to them and say, hey. I wanna do this. What are some good resources? How do you already do some of these things?
I think one of the biggest challenges if when you're getting started is if you don't have anybody that you can kind of go to to see how do I do this thing.
And that was definitely helpful.
And then start looking at what type of things do we have in our environment and what types of APIs are available to them. Do I have to log in via SSH and do screen scraping? That's the other thing when it comes to networking. Right? The support of what's available to access it programmatically is kind of all over the place depending on how old the devices are.
And you're talking about network devices. Like, what kind of switches, routers, wireless controllers, load balancers, firewall? What what do I have in production right now that I need to manage?
Right?
Exactly.
And, of course, this also had buy in from leadership.
I mean, number one, you were sort of hired to do this. So I that presupposes that there is buy in. So you weren't fighting an uphill battle against management who didn't want you to touch anything. Right?
No. I don't think so. I mean, I think there was a little bit of, hesitancy as to what was going to be done and how to do it. But, you know, automation was already a core part of what our IT department did.
Right? And they were very into that. We didn't do that on the networking side. That was gonna be new.
Mhmm. There's probably probably gonna be a little bit of resistance there. Obviously, it was a team I had to work on, that automation was completely new to them. That was probably where a lot of the resistance would come from.
And not saying they're more against it. It's just from a skill set. Right? That's something new for them. What was their goal, though?
Why did they want to bring this, this idea of network automate or automation to the network? I mean, they were already familiar with it on systems, Linux, and Windows. So was it the same goal to just make processes easier? What what were the ultimate goals for you and and your the leadership at your organization bringing automation to the network?
Yeah. So I think it's the same story I hear a lot of places. Just the speed of business. Right? There's a lot of things that the business wanna do.
We do a lot of statistical analysis in the industry that I work in. Mhmm. And a lot of that is doing creating pipelines and doing things in the cloud. Mhmm. Right? And so how do we build those products and tools internally quickly to get the business objective completed?
You know, and I think a lot of what had been happening, especially on the cloud side, is taking the sort of the traditional method of, oh, okay. We gotta create a new subnet, add a route here manually, pipe this VLAN over here. That doesn't necessarily it does work in the cloud, but it doesn't really kinda fit into the the model that we had seen of just automate and sort of create environments that were sort of self serve to our I see. Data engineering people.
Yeah. Right? To just be click they wanna build a new environment to test something. Right?
So how do we take the network roadblock out of that to help them and the business get what they want to, achieve done quicker? I think it was part of the the the goal.
Yeah. It sounds like the speed to deliver new services, specifically on the network, at least the network to accommodate other services being delivered. Right? So the speed, the agility, the efficiency there.
You know, one thing that I remember hearing years ago when everybody started talking about network automation, probably 2015, 2016 or so. Right? It was like every other blog post. One of the things that I heard often was, you know, automation's awesome because you can automate the boring things so you can do more interesting things at your job.
I I did a about a year and a half network automation project at a pharmaceutical company, and that was not the case. Like, it wasn't automating away the boring tasks so I can do the interesting work because I was still at work. It was automating, like, the repetitive tasks and the, the tasks that I maybe were mundane. Sure.
And I was doing other boring work. I mean, I ended up having to manage this, you know, this repository of scripts and and and inventory and other kind of work. So it to me, it wasn't like just making my life more interesting.
It was about making the business more agile, making the team more efficient.
One of the things that we really wanna do was was was, reduce the incidence of error. So, like, when you're assigning VLANs or move you know, moves ads and changes, kinda like, you know, easy stuff that you're doing on a routine basis. We wanted to automate a lot of that stuff away, and and reduce any kind of errors like putting the wrong VLAN on a port than having to track that down.
Automating the updating of documentation. You know, I'm you know, one of the things that we started with, at least that I started with, was actually automating collecting information. Like, we weren't pushing config right away because leadership was scared of that. I was scared of that.
How how about you? What were the first few things that you wanted to automate? What were, like what would you consider low hanging fruit, as you got into this with your with your current organization?
I would definitely agree with gathering of information.
You know, one of the things starting in a new environment, you gotta, like, figure out where everything is. You know, of course, everyone has great documentation. Right?
That would be, in air quotes and Yeah. Sarcasm if you didn't catch that on the audience side.
Yeah. Exactly. Right? So, you know, there's a lot of documentation that was floating around, but it's all, like, ten years old. I'm like, this this isn't accurate. So there's a lot of how do we how do I easily find this information?
You know, spending a lot of time at VARs, I knew my way around at Cisco CLI to pull data out of it really quickly. But when I came to the AWS cloud side, that was still really new to me. But I know, obviously, AWS calls a lot of good APIs to be able to pull that data. So, a lot of that was pulling information down and kind of compiling it together.
Right? So for example, we had a situation. We have a, like, a security appliance in the cloud, and to get from the Internet into the appliance, there's, like, two different NATs that have to occur. So there's multiple IP addresses in the path.
So looking up at one GUI, looking up at another GUI, and then another GUI, and I can't remember IP addresses in my head for more than, like, ten seconds. So having a tool that could just, like, pull all those things together and just show it to me was so helpful. And that was a lot of I I think what I did initially was a lot of just tools to pull stuff from different sources and put it together so I can, like, okay. This makes sense.
I can see how this works now. Yeah.
You know, you'll follow the path in the data center is all all the VLANs on the right interfaces. Right? Like, pull that stuff together. I'm like, oh, it's missing on this one path. Okay.
You know, those are the things I feel like that you're not automate automating away the boring. It's just, like you said, the repetitive, like, okay. Running the same show commands over and over again on four different devices that I've done this a million times. It's not fun.
What's fun is fixing the problem. Right? So I can gather that information faster Yeah. Then I can really dig into it.
And that troubleshooting isn't necessarily the thing that we're fix we're we're automating away. Right? It's the mundane information gathering you have to do all the time, pulling that all together. Mhmm.
Kinda reminds me of that. I think I saw the presentation at, I think it was, AutoCon Zero and at, Nanog from somebody that worked, I think it was at Apple, talking about how they use automation to do, troubleshooting when they have to open cases for vendors. Right? They always want the same outputs, right, from bunch of different devices.
Yep. Do you wanna do that five or six times every time you hit a case, or just run a script that just gathers for you in, like, ten seconds so you can upload it, so you're not sitting on that tech case all the time. You know, things like that, I think, is definitely a good start. I always recommend to people Read operations are probably the easiest way to get into automation.
There's a very small likelihood very little likelihood of breaking things, not impossible, but very small.
Right.
Yeah. That makes a lot of sense because, I mean, for a lot of folks, there's a, aversion to change. Right? And that's understandable. I don't like change as much either. There's also just, like, a fear of the unknown and the fear of, like, breaking things. So if you start with something that has a little impact or little potential for, breaking the network, gathering information, doing show commands.
May maybe the only kind of change that you push your interface descriptions or something. Right? I don't know. Putting new circuit IDs on interfaces. I'm just, you know, off the cuff here.
That's there's still a lot of value there. What you were just talking about are, like, the initial steps of troubleshooting problems. It might be a mission critical troubleshooting workflow or process that you're that you're going through, and you and you're expediting that. You're getting to a you're reducing your mean time to resolution by adding this, automation component to that.
And sure. Yeah. Later on, you can do some much more complex things, and you can tie, various scripts together into an entire orchestration platform. That's awesome.
But there's no reason to, you know, be, overwhelmed by that end goal. You can start with just the basics of, like, pulling information and and, like you said, automating gathering information from show commands. For for you personally, for you and your team, what kind of devices were you working with? Are you talking about traditional devices like Cisco routers, switches, things like that?
Yeah. I mean, we're a bit of a multivendor, shop, but primarily Cisco.
But we do have some other platforms out there as well. But even when it comes to Cisco, right, there's different platforms out there. They have iOS, iOS x e, NXOS, FTD, ASA. They're all very different. Right? So in some ways, just because it's Cisco doesn't make it any easier.
You know, we have other firewall platforms, other security platforms out there as well. You know, dealing with a Cisco ASA versus an iOS x XE device from a, you know, programming, you know, automation standpoint, they might as well be a different vendor, you know, because they're different. So I don't know that that helped or hindered me in any way. It just made it more challenging where I think initially, I would say, alright.
I'm gonna focus on this particular platform that I'm gonna get the best best bang for my buck. Right? Where do I where do I spend a lot of my time before I'm developing any kind of automation or even just, you know, tooling, where do I spend most of my time where I don't want to? You know?
Is it a firewall platform? Is it in our data center platform? And then start there.
Right. That makes a lot of sense, actually.
So let's talk about tools and methods and languages and all that kind of thing. What you mentioned a couple already. You mentioned Python and But what what were you using, and what are you using now, in your network automation journey?
So I'll say back in my VAR days, I I got started with expect scripts, which at the time, I thought were, like, super high-tech, fantastic, and come to find out that's kinda, like, you know, the the, the good old days, I guess.
Right? But, you know, in that particular scenario, it did what I needed. You know, you and I both know from work working there. We'd go to a lot of different customers.
A lot of them were smaller, like, kinda k twelve schools that maybe not didn't have a lot of in-depth, networking knowledge. And I would bounce from one of the other. And, obviously, with with their permission, I would run scripts that would basically pull all the operational and config data down so that when I wasn't there and they called me up with a question, I could look it up and say, oh, yeah. You have this link between this switch or these VLANs over here and be able to, you know, help them out, which is always great for customers to just be able to answer a question real quick for them.
You know, moving into where I am now, you know, I took that to you know, move that into the kind of the the modern version of that, which is, I guess, would be Python. Right?
And just learning what to do with that. Although, I would say if I had to go back and do it again, I probably would have started with the Ansible. I still don't do a lot of Ansible. And now knowing what you can do with that without having to know a lot of programming, I think it's pretty powerful. Yeah. It has its own syntax that you have to learn, but you can do a lot without really understanding Python and having to write raw Python code.
Mhmm.
But I start with Python and just kind of, you know, I was doing simple, you know, requests against web APIs because we had a lot of tools that do had, you know, web based APIs, which a lot of tools do now. And then getting into obviously, you deal with a lot of Cisco devices, you're gonna have to deal with SSH screen scraping. There's just not really a way around that. Right? So Yeah. There's different, Python libraries out there for that.
The two that I've used the most were, NetMico, which is a pretty popular one that just supports a lot of different vendors, not just Cisco. Mhmm. And then another, library from Cisco called PyATS that has, a parser built into it for doing CLI parsing. But, again, it's not just it it's made by Cisco, but they support a lot of other tools as well. And Py ATS does a lot of stuff besides just parsing, which is why I've kinda leaned mostly these days into into using that. Again, being a heavy ish Cisco shop, it supports most of what I need to do. Mhmm.
You know? And then from beyond that, the next thing as well I found is once you're trying to write code, you need a way to organize it because I'm I'm sure a lot of people start with they'll write something in notepad and then run it, and they'll save it, modify it, save it again, then you break something and you're like, what? It it worked earlier today. Right. How do I get back versus just undo, undo, undo?
I think get has been super helpful in being able to sort of take a version of the code you have, play around with it, and see if it works, and then commit it or roll it back if you need to. And, again, that was a tool that, luckily for me, was heavily used by some of our other teams. So, I had never used Git before, and I luckily had people that I could just go to day in and day and say, hey. How do I do this?
Or how do I do that? And then, you you know, they give you the right pointers, then you can kinda figure it out from there on your own, find a lot of resources online. Git is pretty popular. Right?
So it's easy to to kinda Google stuff, on there as well.
You know, and then the next big thing I I feel like once you get into things like Git is how do you organize all your scripts?
Because the one thing I've run into and I think a lot of people deal with is I wrote something like this before, but in my fifty something scripts that are in different places, how do I I feel like I spent I used to spend a lot of time just trying to find where I wrote that the last time to pull it back in.
And trying to figure out a way to organize all that, I think, is, that's its own area of study that's kind of interesting to me Mhmm. About I think I've learned from a lot of people in the programming well about monorepos versus, I forget the term, but they have a lot of individual repos and which is better and which is, you know, Yeah. Use case. It all depends. So Mhmm. I found myself getting more into these other niche areas of programming as a result that may not have any use case in some of what I do, but it's just you know, I always find it good to pick up different things about IT that you may never know that you're gonna need as well.
Yeah. Absolutely. And then it's use case specific. I mean, it's it's gonna be your organization.
Do you have specific functions and modules that you guys are using often that end up finding their their their way into Ansible playbooks. Right? Right. Whereas another organization may not.
And so, you know, the as much as we want to automate and systematize and make everything very common and, repeatable, you know, like Greg Farrow back in the packet pushers always used to say, maybe he still does, that every network is a slow snowflake, and so there are those, customizations. But that's the whole point here is that we are trying to automate, as much as we can with the expectation that it's it's not possible to automate one hundred percent. I was at a network user group, networking user group out in New Hampshire recently and the and the speaker there said that, you know, your your goal is like eighty percent.
You wanna automate away about eighty percent is of of the activities, with the understanding that it's just not possible to automate that last twenty percent. There's weird stuff and one offs and things that happen that you have to get in there and, and figure out and, whether it's at the CLI or or whatever other mechanism you're using. But, but that's still progress. You're still automating away eighty per or not.
I don't wanna say away. You're still automating eighty percent of the tasks and activities, whether it's gathering information or all the way to pushing config and more complex workflows. And that's still a huge step forward.
And so what were some of the roadblocks that you faced? I mean, you're dealing with predominantly Cisco devices, though there were some others, and you had multiple versions of code and different platforms to deal with, operating primarily with Python and Ansible that you mentioned, although you did you did say expect expect scripts as well.
What were some of the roadblocks in your in this current role that you're in right now?
You know, I think that the biggest roadblock for me is just kind of, you know, learning this stuff and having a platform where I could play around with it. Again, luckily, I was in an environment where we already have a lot of automation in play on the Linux and cloud side. So I spent a lot of time working with those teams, understanding what tools that they had and how they use them to get familiar with them. Like I said, we have a local Git based repository system.
You know, I had to go create something. Being in a heavily regulated industry that I'm in, using new tools is always, a little scary to some people. So having that definitely helped Mhmm. Quite a bit.
But, you know, one of the things was, okay. Well, I need to be able to put password in for network devices for these scripts to run. Yeah. I can't just say that in a script.
Script. You don't wanna save that in a repository, obviously.
You know, how do I do that, and how do we make sure that the character, you know, if there's a special character that needs to be passed in that it's escaped correctly. And a lot of those little things are always seem to be kind of the initial roadblocks, and you you just gotta work through that because once you get past it, you're pretty good. But, again, having a being in a heavily regulated industry like I am, obviously, there's a lot of concern just using new tools. Right? Luckily, we had a lot of that in place.
And I think one of the things that was super helpful helpful for me was being able to have a knowledgeable group of people that I could go to. Yeah. Right. You know, if they were all knowledgeable in Python and I said, you know what?
I wanna write all my scripts in Go. Well, that's probably not a good idea. Right? I mean, I could do that and have to suffer through a lot of it, but we already have all the frameworks in place for dealing with Python, dealing with secrets in Python, storing Python code.
We have knowledge around it. I I'm gonna go that way. That definitely helped through a lot of those types of roadblocks. Yep.
You know, the other one obviously is probably cultural. Like I said, you know, my position was created to do some some, automation, but the people that I worked with in networking were not automation people. And I could create all the great automations I wanted to, but if they're not gonna use it, it helps me, but it doesn't help our team. It doesn't help the business at all.
Right? So how do I create something in a way that's easily reusable for them and packageable in a way that they can easily use it, and get the value of it without having to learn everything there is to know about virtual environments in Linux and Python and how to create packages and package dependencies and all that kind of stuff. You know?
That was probably a big thing. It's like, I create these things, but how do they use them? And it's it's still a thing I feel like I struggle with a little bit. Right?
I mean, it's not like I have a perfect answer for those things. I'm still trying to work through how do I get these other guys to be able to use it, but not have to spend the time that I've spent the last, you know, five, six years learning all this stuff. Right? Yeah.
Yeah.
I'm I'm looking over on my other screen of the show notes right now, and I'm looking at a question about what the real reason traditional network engineers, traditional quote, unquote, right, traditional network engineers, whatever that means, should move in this direction and adopt, network automation as a practice and maybe even a philosophy.
I mean, I mentioned my ideas here, but why do you think that a traditional network engineer turning a virtual and physical wrench should move in this direction?
I I think that pretty much everything else in the IT world is going this route of being able to quickly deploy services via automation. Mhmm. And if you wanna sort of be a part of the solution, I guess, being able to get your end of it involved, I think, is is a big deal in my opinion. Right? Do you wanna be the roadblock, or do you wanna be the enabler?
You know, one of the things that's worked really well for me is being able to, you know, facilitate new ideas and projects with people, being that expert. If I can take that knowledge that I have, apply it to this project, then get it done quickly and move on to the next thing, you know, I get to work on more exciting projects, do better things that the the business gets things deployed faster. It's not just automating away the boring stuff. Right?
As we said, that's kind of a a fallacy, I think. But it does allow us to do other things that are more exciting with the network. Right? Like, do you wanna spend your time creating VLANs, or do you wanna spend your time creating cloud infrastructures that help run brand new applications that the world's never seen before.
Mhmm. Interesting. Yeah. Yeah. So, ultimately, there are submits. You know? We're not we're not just trying to, automate away all the boring activity to make our lives more pleasant.
It's it really is to, accommodate the business, delivering services quickly.
And I would also add, one of the goals that, you know, that we had, at the organization where this was kind of a big a big initiative of ours was, eliminating human issue human error, eliminating just problems, reducing problems in the networking, making making things more consistent, and and automating certain things like updating of documentation whenever something changed, it updated an inventory file. You know, that kind of stuff. So that way we had more consistency and, and and things were therefore more quiet, more reliable. I mean, ultimately, that was the goal, was to never think about the network because it was just that rock solid and that, you reduce this idea that if I wiggle the wrong wire, you know, everything would blow up.
Now, of course, you can never really automate everything away. And I see that you put that in the notes. So can you explain that, like, this myth that, well, we can automate everything. Well, can we?
Can't we?
I mean, I suppose you could, but some of it's like, what's worth your time. Right? There's a lot of things that I do that are like one off tasks. Like, I could just do it and get it done in half an hour, or I could spend three hours writing a script to do it.
I see your point.
Right? Like, does that make sense? No. It probably doesn't. Yeah.
You know, and to in in my opinion, automation is good for repeatable things that you are predictable Yeah. To a certain extent. Right? If when you get into programming and writing, you have to the logic can condition stuff that you have to figure out. Like, if it's a one off thing, like, you're not gonna know every possible outcome for that.
You know and, again, what's we if you're really good at the CLI, like, I know I'm pretty good at I can type out shorthand commands in iOS really fast. There's certain things that's just faster for me to do that way. And I I often think of myself, I should write a script to do this, but I'm like, but I can do it faster this way. Yeah.
So it's just what is worth the time and effort, I guess, for certain things. Whereas, like, I need to deploy a new application and we do all the time, and it's like I gotta add a VLAN to twenty different things. Okay. Well, that's a pretty good use case for automation.
Yeah. That makes sense. Opinion.
Right. Right. Adding some new, prefix lists, some new ACL. All those things can kinda get plugged in, create your templates. It draws from whatever sources of truth, your IPAM, maybe updating DNS, all those things. They're very, very like, they they're the things that happen every time you put a new application into into production. Right?
And another use case actually around that that kinda pops in my head that I think is is pretty useful for people that I've seen, Jeremy Shulman talk a lot about in some of his presentation about what they've done at Major League Baseball, which can be quite intimidating if you're getting if you're new into network automation. But this idea of testing, pieces of the network quickly. Mhmm.
I'm sure we've all been in the case where back in the in office days pre COVID where something something would happen and you get, like, fifteen people saying right in your cubicle saying, is it the network? Is it the network? Well, what do you usually do to check that? Run a bunch of show commands all over the place. It's like, well, these things look okay.
What if you could just run a script that just checks all your interface counters, all your CRC to like Yeah.
Yeah.
You can do that in, like, fifteen seconds and say, yep. These look good. I mean, it's not everything, but it helps some of those things go a lot quicker.
Yeah. Right. Right. It's a starting point.
You mentioned, Jeremy Shulman, as a resource, somebody that you've looked to out in the community, out in the industry who's doing some work in automation and putting some content out there. That's awesome, and I encourage folks to look him up.
What what are some other specific resources that you can or that you utilize and that you might wanna point others to, whether that's blogs, podcasts, textbooks, you know, what whatever it happens to be?
Yeah. There's definitely a lot out there, which is great. You know, ten, fifteen years ago when I started in IT, some of this stuff didn't exist. So, you know, being a big Cisco shop, Cisco DevNet has been a huge, resource that I rely on quite a bit. They have a lot of, sandboxes. So if you don't have a lab environment, it's not it's a single device, but you can still practice writing something to a device without worrying about blowing it up, you know, a device in production because not many of us may have a spare, you know, NXOS, data center switch line. Right?
They have a lot of good videos out there. They have a lot of, YouTube series out there that have been super helpful around Python and network automation.
Another good place for if you're getting started is Kirk Byers, has a lot of different, Python courses out there for network engineers.
He's also the one that, maintains NetMico. He's got a lot of experience. He also has classes in, like, things like Ansible and and whatnot. So it's been a huge resource. Some of his classes are free too, so it's hard to beat.
John Capobianco, who's over at Cisco now, I remember hearing him originally on the Packet Pushers episode. He's done some fascinating stuff and really likes to document a lot of the stuff he does on YouTube and blogs. He's a great resource, as well. He's got a lot of great sample stuff out in his GitLab as well that I've looked at as well that's been pretty, pretty helpful.
You know, one of the things you'll find with him and other people that create Python libraries is if they're open Mhmm.
Yeah.
In, like, GitLab, you can go look at things and say, oh, that's useful. I could I could use that somewhere. Let me pull a piece of that for something. I've definitely done that with a few of this documentation piece of where you could pull stuff out and generate documentation from it.
Network automation nerds podcast on packet pushers is also really good. I've been listening to that even before they were on, Packet Pushers.
Mhmm.
A lot of interesting guests on to just get different perspectives on things. Right? Like, one of the things about automation I think is interesting is that if you only approach it from a networking standpoint, I think you're really limiting yourself. Mhmm. Being able to, you know, look at other areas of automation, how you do things and manipulate data is kind of important. And he's, had a lot of guests on there that kinda talk about different areas of automation. It's it's, pretty interesting.
And, obviously, virtualization tools to be able to virtualize network devices, I think is key. Right? Like, as we said before, having physical devices is great, but if that's all you have and those are your production devices, it's probably not where you wanna mess around with with automation. So, I've used Cisco modeling labs a lot. GNS three, back from my CCNA days, I think that's still around. And, even geo lets you run a lot of, multi vendor images as well. So you can kind of create topologies and interact with those, with automation tools.
And then now you can also tear it down and build things really quickly as well, which is super helpful.
Yeah. Absolutely. And I will say that GNS three can be very complex and super powerful.
So I would I was using it all the way into CCIE studies, not way way past CCNA days and still a very, very useful resource.
So I appreciate all that that you, just mentioned tons of resources out there and so much of it is for free. So for folks that are interested in getting interest, in into network automation, there the the barrier of entry is very low. You don't need to go out and spend twelve thousand dollars on a boot camp. Those boot camps are awesome.
Don't get me wrong, but you can start with gathering information, some basic scripts, off box scripts, not TCL scripts, and and, and start from there. And and leverage the resources that are out there on places like YouTube and, and and other free resources like that. So, Steve, this has been a really interesting conversation. I love getting into the weeds, of, of networking and today, specifically network automation.
Thank you for your experience and, and, you know, just sharing your knowledge with us. If, if folks are interested in reaching out to you with a comment or a question, how can they find you online?
I'm not super social, but I'm definitely on LinkedIn, so you can find me on there. There's a couple other Steve Leepers, but only one of them that I know works in networking, so it should be easy enough to find.
Okay. Fair enough. And you can find me still on Twitter at network_phil. Still active there. You can search my name on LinkedIn.
And, if you have an idea for an episode or if you'd like to be a guest on Telemetry Now, I'd love to hear from you. You can reach out at telemetrynow@kentik.com. So for now, thanks very much for listening. Bye bye.