We have a very special episode for you today with our good friend Scott Robohn returning to the show to discuss how industrial engineering and the concepts of Edward Deming and operational efficiency normally discussed in manufacturing circles apply to our niche of technology, namely network operations?
This is an interesting one in my opinion, so let's get started. My name is Philip Gervasi, and this is Telemetry Now.
Scott, welcome back to the podcast.
Really appreciate you returning as a guest. I mean, I consider you a friend of the show and, and enjoy having you come on. And I and I really value your insight, because of your interesting background, not only as an engineer, and as somebody who's got a lot of business acumen, but also because of what we're gonna talk about today as far as, you know, your educational background, formal educational background from years ago. I won't say how many years ago, but let's just say x number of years ago, that I really think applies in a very special and unique way. It's a unique perspective that I think is very, very relevant as we're gonna discuss today. So now that I've buttered you up, and stroked your ego, why don't you introduce yourself for the audience? And and I would like to hear a little bit of detail around that, background.
Sure. Well, I I feel like now that I've achieved friend of the show status, at some of the next level in my career. So I feel like there needs to be challenge coins, you know, for the Challenge coins. Now friend of the show.
Alright. So, well, you're generous. Thank you for that. Yeah. Let's take a little different tack today.
Talk more about, some things that are adjacent and around networking, but we'll try to connect some dots. So academically, you know, like, we'll just put years to it. I graduated high school in nineteen eighty four in Syracuse, New York.
I was not a really I did not have a lot of guidance on what to do in college, and I only applied to two schools.
And I liked math and I worked with computers in high school, so I thought engineering would probably be a good way to go.
I ended up at RIT and started in the computer engineering program, which was, you know, hybrid of computer science and their microelectronics program, which was one of the first schools in the northeast to have a micro e program where you can actually go in a clean room in a bunny suit and learn how to make chips. But it was not three nanometer technology in nineteen eighty four.
I, I lasted one term in that program. I, I had a wake up call with the department head, in a very, very challenging intro to computer engineering class. And I just kinda landed in industrial engineering, and, I won't give you a full disposition on, everything industrial engineering entails.
Some people jokingly refer to it as imaginary engineering or the liberal arts of engineering.
It I never had to take a circuits class and, I avoided some of the harder mechanical engineering classes as well. But there's real, focus on process and structure. So I had a lot of statistics, some really interesting classes on material science and what happens when you heat metal and bend metal over and over and over, and how things, look on a manufacturing shop floor.
One key concept that I kinda caught on early was was workflow.
You know, when you're arranging the assembly of complex machines and need to have multiple parts come together in certain sequence to create an assembly, you really have to arrange what gets made when, where it gets put together, and what those assembly times require and any other material transit times and so forth. So, if you fast forward a little bit, I then landed in grad school, in a in an offshoot of industrial engineering. I actually went to Penn State, and was in a program, called Operations Research, which is really kind of frustrated applied math.
But it's about mathematical optimization for manufacturing problems, scheduling problems, and so forth. So I kinda had this, you know, systems view early on of how things come together.
It's actually where I was first introduced to Dijkstra, for scheduling trains and planes well before I knew anything about networking and shortest path first.
Okay. So Yeah.
So first foreshadowing of, eventually landing in networking.
Okay.
Yeah.
So the shortest path first was a literal path involving railroad tracks. Interesting. Yeah. And I'm glad you said systems and and workflows, but I don't see the connection because we have no workflows or systems and operations. Everything is just willy nilly ad hoc fly by the seat of our pants, isn't it? Like, hence today's podcast.
Yeah. No. I, and I, and I think that comes to like the strong, cultural heritage that we have of network engineering. And we've talked about this a little bit before, but, you know, kind of that incredible growth, you know, not just in the nineties and into the early aughts, but even during the development of, ARPANET, right? It took very creative people to take microcomputers and shove serial cards in them and turn them into IMPs, you know, internet, internet, message processors. I don't think I have the acronym decode perfect there, but, you know, there's always been an emphasis on doing and getting it done, which is creative and innovative and awesome.
I think the, the growth, you know, once your, once your grandmother got her aol dot com email address and, dial up internet was no longer acceptable to her, You know, how much was bandwidth was enough? Just a little more. And, you know, that whole culture of we got to meet this demand and build it, build it, build it, focus much more on getting things up and running versus thinking about how we do this in a sustainable way. Does that that resonate with you?
It does. And so what you're saying is that there was a time when things were less complex, specifically in IT and network operations, such that, though not ideal, we could get away with flying by the seat of our pants a little bit more and without this systems thinking. But that's just not where we are today. And I'm I'm gonna guess now because, you know, you reading between the lines, because of the complexity and the pervasiveness of networking technology and how it really does underpin everything that we're doing in our personal work lives, you know, cultural, all these kind of things. Is that is that right?
I I think it's notionally right. I would say it is a little better than, you know, the cowboy days of getting those first t ones installed and that first one gig circuit.
You know, there are a lot of business operations that obviously depend on Internet connectivity and that drives some stability in some process, but I still think there's that underlying we we don't have the rigor applied to network engineering that probably we had for manufacturing engineering or that people like W Edwards Deming, and others tried to drive in in those environments. And I'll draw just a little, you know, contemporary analogy.
I think we see some of that in the whole GPU craze now, right? Where the thirst for, acquiring GPUs and building clusters, you know, there's a lot of just get it done, but at much higher budget amounts. Right?
I'm not saying the two situations are exactly the same, but I I I feel a similar fervor for building out, AI model processing infrastructure as the fervor around building the Internet thirty years ago.
Interesting. Okay. So you mentioned, from a high level what industrial engineer and you did say that, like, there was some math involved. There was.
And and certainly some, some traditional engineering, in, you know, in that sense. But you really spoke to process and systems thinking more than anything else here. So, let's drill down a little bit more specifically. What are some of the what what are some of these individual concepts that you learn in industrial engineering that, we should start to because I'm sure there are very, very many.
But what are some that you'd like to call out that we can start to think about in the context of network network operations?
Sure.
And and in no particular order, but I'd like to get some, detail now.
Well, I think the concept of workflow is fundamental.
And I think with the conversations that I've had with all sorts of organizations over the years, It doesn't matter whether they're carriers, small enterprises, and everywhere in between.
There just is more emphasis on, okay, you know, Phil, you're in the seat. You just need to figure out how to get it done. Nobody's gonna tell you how to do it.
And there's there's there's freedom in that. Right? But that means everybody else who ever ends up in that seat, needs to figure it out for themselves too. So thinking about the process of what do I do, you know, to upgrade the software on this router?
What does it take to put a circuit into service and authorize it for use, and put it into your routing protocols?
Like, thinking through these things from a process perspective, is something that we don't, we don't put as much effort into as I think we should. And I'd say there's a corollary here to automation.
You know, we talk about, how how can we be successful in implementing an automation strategy?
Well, along with finding the lowest hanging fruit, you know, starting with the things that take the most time repetitively, templatize them, templatize your configs, make it repeatable. And I think process and workflow is the analogy to how a how a process gets executed to templates for device configuration so they can be more easily automated. That's a that's a real connection that I see.
Now you're talking about the process of doing. You're talking about this idea of, planning and approaching things in a very prescriptive manner. Excuse me. But from an operational perspective and how to do this thing.
When, going through some of the research yesterday for today's episode, it also seemed that there was an element of, being able to get a handle on the processes of the actual manufacturing process itself. Mhmm. The the the widget going through the assembly line and all of the the different components and how to understand where there are inefficiencies in the in the in the manufacturing process itself at different points. Right. How to you know, there was a element of observability that I saw there and, and make that more efficient, reduce error, and and look at it as a system, that whole thing. So in terms of of less process and more the packets going through a system, that does speak a little bit to architecture and design to me as well, not just upgrading code on a on a core switch. Right?
Well, so a perfect analogy to that for manufacturing is is quality control and statistical process control where, you don't, you don't want to just be reactive. You don't want to say, okay, this is in service now, how's it looking And just measure quality after the fact you, if we, if we use the manufacturing analogy, you know, if you're, if you have something wrong in your process, that's creating parts out of spec, you're wasting material, right? And that's costing you money and wasting time and resources.
You do, you do want that observability on your manufacturing process to have that closed loop feedback, but you also need to think about, well, how do we build quality into the process from the beginning, and be proactive about it and not just reactive?
And and this gentleman, w Edwards Deming, who was kind of a luminary in manufacturing engineering and statistical process control, you know, from the forties all the way into the early nineties, you know, he was very much you need to build quality into the process early. And what that means for NetOps is, you know, thinking through what are those organizational connections need to look like? What information do I need, you know, from the service ordering team to do the certain thing in the network to put the switch into service, circuit into service, you know, what SLAs are associated with that, and thinking through that across organizational lines and having kind of, like, a an orchestrated workflow. I'm using those terms very intentionally.
And it seems like you're speaking a lot more to organizational behavior and not necessarily the behavior of my router. Although you did mention observability and and and understanding, how the system is functioning, even apart from the reactive methods that we use. So just collecting flow and just collecting streaming, right, from my devices. That that's that's, re reactive. That's what already happened. The existential already happened because it might have been, you know, a moment ago, but it's nevertheless the past.
So how do we how do we do that proactively? And we'll talk about that. But but you have been talking thus far about organizational behavior, like, in like, people, like human beings, not not how latency looks on my on my devices or between two links or something like that.
So let's get into that a little bit. You know, you you mentioned Edward Deming. Can you give me a little color on who he is and and what are what are some of the, concepts that he put forth that we should be applying and thinking about?
Yeah. So so, again, Deming was an academic and a practitioner in manufacturing engineering.
And, you know, I won't be able to do his biography justice here on short order, but suffice it to say, I think he worked for eventually the US Department of Agriculture and ended up getting involved in, efforts in Japan to help rebuild their, manufacturing infrastructure after the devastation of World War two.
Oh, okay.
And he, you know you don't no one person can take credit for for any any big thing like that, but he certainly played a role.
And his his constant commentary was all the concepts for statistical process control, plan, do, check, act, you know, things that he, developed and borrowed from others in the US. He brought that all to Japan that had incredible improvements on their manufacturing quality, which is demonstrable from the fifties into the sixties where, you know, Japanese cars really started becoming high quality Yeah. And and more popular.
You know, and, obviously, we have Honda and Toyota and others today that have really benefited from that. Mhmm. Yeah.
But he was a little, miffed would be the highly technical term that, you know, here these these concepts that, you know, we put together are being taken, taken to heart in other parts of the world, and American manufacturers really weren't paying attention.
Oh, okay. Interesting.
That was that was one of he, I hate to use the word rant, but he ranted about that quite a bit.
Wrote Yeah.
Wrote a lot about stuff like that.
Can we go back and define what those two terms are that you mentioned? Plan, do, check, act? I can probably guess what it means based on the, you know, the words. And also, SPC, systems process control, I think you said?
Statistical process control. Excuse me. Statistical process control. Yeah. Can you just define those?
Sure.
So SPC in general, every good industrial engineer learns about, is a method of measuring quality for the output of a a given manufacturing system to understand when an out of spec component is really out of spec or as the result of natural variation in process.
And what you what you wanna use these tools for are to minimize the, variations in process. So you can get things as tight a tolerance as you can, within a certain range, within a range that, you know, fits fits your budget, so to speak.
You know, certain, you know, train train track parts have one set of tolerances associated with them. Microelectronics have another set of tolerances associated with them. But SBC was a process that used graphical, analysis and really control charts to track, the process output quality process, of a given manufacturing line.
Plan, do, check, act is exactly what it sounds like. It's your Right. Your own, you know, human feedback loop. Right?
Plan to do something, actually do it, check the work, you know, be very data based on, you know, what did what did what I do, result in, and then take that and feed it back into the next planning cycle. So it's that iterative, you know, I I think there's also a variation of, plan, do, study act.
Mhmm.
You know, where study is a little more, academic or, you know, a little deeper than just check.
So, but really useful, you know, ideas for lots of things we do in NetOps and beyond.
Oh, yeah. I mean, already, I'm seeing the connections to DevOps principles that we've been talking about for a long time now in network operations and applying that and, hence, the, you know, the new the new term NetDevOps. Right? Yep. And, so we'll get into that a little bit more for sure because I think that's the clear connection here, from a for at least a high level and how that is applied.
And then, of course, how we define that dev ops, that's always a thing. You know? Everybody's got a different definition as well. Right.
But, certainly, it does involve processes and efficiency and this ability to observe the system to see, you know, for lack of a better term, it's health. You know, if it's meeting the the requirements, if it's meeting the expectations, measuring that in some way, and then being able to put that into what in DevOps, you know, our CICD pipeline. Right? Our closed loop kind of, like, continual integration, continual deployment, and, and then and then, of course, all the technical mechanisms involved in doing that, which is I think where a lot of network folks, especially me.
Right? We we love the technology. I think that's where we end up focusing and we get lost in in those weeks. Important weeks for sure, we need to understand which database to use, how do we ingest telemetry so that we can understand what is the state of the network.
And so in that light, when thinking about, like, observing a system, you know, you're gonna see those outliers or those the, you know, configuration drifts or there's variations in the system variations in the metrics, which is something that we need to measure and it's part of kind of our knowledge base of how we make decisions and networking.
But but there is a there's a behavior component to that, like a human behavior. Right? There's also this idea of of being able to understand how this one metric fits into the entire system. It's like a like a technical KPI that onto itself means very little, but what does that mean for for the health of the entire system? So let's let's talk about that.
I see that in the notes, but I I'd like you to expound upon that a little bit.
Sure. Well, let me let me zoom out and kinda use Deming's framework for what he called, and and wait for this because it hits a little funny, his system of profound knowledge. Now in, the twenty first century, you know, we're recording this in twenty twenty five. That sounds a little funny.
I, I think it had a different weight. You know, when it came out in the fifties or the sixties, but, he kind of has four, four pillars that I think are really useful for thinking about any any system of activities, including network operations. So the first pillar is appreciation for a system. Nothing happens in isolation, not one person, not one router, not one circuit, not one NMS. You know, it's all connected.
Even your even your ticketing system, your connections to services in the cloud. Right?
There are implications for viewing things as a whole, and that can be really, really helpful. Even, even a router or any network element is a system. It's got hardware and it's got software. There are different software processes and there are different hardware components.
So being able to think holistically about any network element all the way out to your organization, I think is really, really useful.
Second pillar is something he called knowledge of variation. Right? And that's really what statistical process control is all about.
Making sure you're observing what's going on so you know what variation is happening so you can figure out what variation is normal and what variation you can eliminate.
That would translate in NetOps to things like, you know, mean time to service restoration, mean time to innocence, Right? And having metrics on, okay, what happened during that outage?
What what did we have no control over, you know, variation that that is natural, versus what could we have done better, more quickly, more effectively? Right? So, measuring and being very data based, he follows that into his third pillar, what he calls the theory of knowledge.
You should be making decisions based on data. Right? And and Mhmm.
In common terms, we would call that being data based or data informed.
Okay. He also put the idea of small experiments in there. Don't try to change everything in across a whole system at once. Do a little change. Measure the results. See what happened.
And there's some really great examples of that from, like, small batch manufacturing, that can lead to much, greater process improvements. You you can control things much better in a smaller experiment. Right. Right. And then his fourth item, fourth pillar is what he just calls psychology.
That's basically understanding human behavior, right? How people work, how people interact.
We're not just Python scripts. We're not just iOS commands.
We're not just tickets in ServiceNow. Right? There are people interacting with these systems and with each other, and that's an important part of the whole, of the whole system's view of things. Mhmm. Now on the people pieces, Deming puts a lot of responsibility on management and leadership.
For example, you know, he he would say quality happens in the boardroom.
Right? Decisions about process are made at the highest level, which means you can't blame the NOC engineer or, you know, the, the escalation engineer or the network architect.
There should be processes defined with quality in mind, from day one.
Of course, there's always we have to be watching for, you know, our people not understanding process. If they are understanding it and they're just not following it, there are, you know, actions and remediations that need to take place. But he does put responsibility squarely on the shoulders of leadership, and I I think that's accurate and, and important today.
Let me I'll pause and let you comment there. I can keep going.
Well, that is probably one of the more difficult pieces of this whole thing because I can study syntax.
I can understand LSAs, and I can, you know, I can scan an ACL and see where there's a rule that's blocking my traffic. Albeit, those things might be difficult at times, and they get more difficult as complexity increases.
But people are hard. Yeah. You know, trying to affect change organizationally, when you're talking about human beings and egos, as you mentioned. You know, we we jokingly talk about the layer eight issue.
Right? Human beings. And then there's the layer nine issue, which is the the politics and budget. Right?
Is there a layer ten? I don't know.
But if there is We could define it right now.
That's right. It's probably involving people. Right? Yep. And so, it's it's interesting that we you know, we're talking about network operations.
We're talking about industrial engineering, which sounds heavy and scientific, but it really does coming it comes down to organizational behavior. And literally, you mentioned the word psychology. Right? Understanding human behavior, motivation, you know, the the the system of of rewards and punishments, not involved we're not talking about reinforcement learning.
We're talking about human being, type of reinforcement learning.
Yep.
So, it's very interesting that we're going here. And I again, I do see the tie to DevOps where we always talk about DevOps and NetDevOps as a as a culture and not a technology.
Yeah. Well, so so there's a couple other things that Deming was big on that have influenced DevOps and I want to influence NetOps even more with.
But sticking sticking on the personal theme, you know, just as an aside, this is a really kind of a full circle thing for me, you know, because I learned about all this stuff in the context of manufacturing and, really rediscovered it thirty plus years later and seeing primarily through DevOps how some of these principles actually influence DevOps. And now what does that look like applied to NetOps? What is NetDevOps or DevNetOps or DevNetSec super duper ops?
Sorry. We'll we'll we can refund that later. But, it's just been a really interesting exercise for me and to kind of work this into some of the ideas I've been talking about via total network operations.
Yeah. I was gonna bring that up, what you're doing now with TNOPs for sure.
But yeah. So other people issues that the Deming really championed was that continual learning, you know, have mechanisms for people always be learning about how they can be using new tools and doing better, you know, in the positions that they're placed in, driving out fear and encouraging open communication.
You know? And this is nineteen fifty. Right? Today, we kinda think, oh, of course. Of course, we should be driving out fear, and promoting open communication.
I'm not sure it's any less of an issue today than it was in the fifties.
I agree with you one hundred and ten percent. I think a lot of a lot of what we hear, and I'm sorry to interrupt you midstream, but a lot of what we hear is probably lip service and and a lot of corporate speak about, let's break down the silos, and then there's really no silo breaking down. But if we could, even a little, I think the benefits could be profound.
I I I totally agree.
Another thing that has always hit me, with him is this idea of, removing barriers to pride and workmanship.
Like, think about the last time in any job that you had, where there was any emphasis on pride in what you do, like the result of your work.
That's pretty vacant for me. Like, it hasn't really been an emphasis of anywhere I've ever been.
Now for for creatives in particular, like, they could see the time much more directly. Right? You know, I created this piece of art, you know, or I created this incredible structure.
And being able to have pride in that is maybe the dots connect more easily.
Yeah, I designed this organization or I designed this process, or I resolved that circuit issue in, four minutes and seventeen seconds.
You know, there's a place to actually be proud of the things that you do there and being able to trumpet that, in the right way. Yeah.
So the it's it's an element I think we're lacking today.
So you've never been so proud of the artistry, of a Visio diagram and how it's okay.
I I I'll give you a very specific example.
In the in the nineties working for a carrier, when we were trying to run OSPF over an ATM network, I wanted this one picture that aligned my ATM PNI boundaries and my OSPF areas.
And you know what? They're two completely orthogonal disconnected things.
And trying to put them on one picture was an exercise in futility. Like it didn't matter. It really didn't matter. But I was hung up on, I wanna have this unified view, you know, of the ATM layer and the IP routing layer.
Yeah.
And, so that's that's tabs, by the way.
Exactly. And Oh. Video or on our spreadsheet, whatever it happens to be. So we can look at layer two, three, four. Yeah. Yeah.
You were I'll dig up I'll dig up that Visio diagram from nineteen ninety seven and, fix it now.
So It it is interesting that you've talked about coming full circle. Now this is a this is an aside. This is not related to Deming per se, but, you know, where you were academically thirty years ago, thirty plus years ago, and then where you are today. And you're seeing all of that now come back to back into play and, and seeing the fruits of that as you're able to apply that to, what you've also learned over the past three decades in networking.
Where interestingly, you know, I have I don't have the same academic background. My background, as you know, was, I was an English teacher, twenty more than twenty years ago, twenty five years ago. And, for a long time, I was focused completely on Cisco certifications and Sure. And, you know, racking and stacking gear and and and putting, you know, ASAs in.
Do we still do ASAs? No. Whatever they are now, but you understand what I mean.
The virtual ASAs still, I think.
I think it's still started. Yeah. Doing traditional network engineering. And then in the past five or six years, having the need to call upon my background as an educator and then as somebody that can, you know, take complex topics and then turn it into something that is digestible and and easy to understand, that has been the at the forefront of my career the past five years and what has propelled me to into different new opportunities.
So and aside from Deming, but, something I wanted to mention as I think a commonality between the two of us where it is amazing how things come full circle. And so for anyone out there listening, you just don't know where ten years from now is gonna bring you or twenty years from now.
So Well, the and the I think one of the factors that plays into that is all you're learning is additive.
Like, you don't have Sure. You don't have buckets. You don't have silos in your brain. Mhmm. You not to reduce a human to this, but in a sense, you are the sum of your experiences.
Yeah. And you know what you learned about Plato or, pick your favorite British author. I don't wanna put, you know, words in your mouth.
You will connect the dots because we're pattern matching machines. It's what we do. And so on on one hand, you know, this full circle effect is awesome. On the other hand, Scott, what do you expect? You've been, you know, thinking about these things they've been brewing, even though you haven't thought about Deming for thirty years.
You know, certain moments help you, see, oh, yeah. This is this is how a systems view of NetOps could really be helpful. And here are pieces that I can borrow, and implement. Maybe it's not a full fit. Right? And let let's bring that into the DevOps versus NetOps conversation at this point.
So, you know, we we've had so many acronyms thrown at us in network engineering over the years.
Please don't get me started about software defined networking, software defined LAN, software defined this, etcetera, etcetera.
I get the principle and I get why, you know, software eats the world. Thank you, Marc Andreessen, but all software runs on hardware and both are really important and they always will be. So, but, you know, we've seen the wave of, of, you know, what we should just be use DevOps for NetOps. It's like, okay, let's think about that. Right.
So there are DevOps thinkers like Jean Kim, like John Willis, who will talk very openly about Deming and, you know, principles that Deming promoted are part of of DevOps, and that's great.
I think the gaps exist in, you know, a software creation process that's really, you know, more the focus of DevOps versus working on physical equipment that exists in time and space and certain connectors go in certain ports. They're not fungible.
And you've got to accommodate that. And the way I rationalize this, like, I'm I don't come I I come not to, bury DevOps. I've come to praise it.
Sorry.
You know, they're really like the whole idea of bringing developers and operators together, you know, less organizational distance, that's great. I and I think that's very Deming esque.
I think the way I like to view it, I wanna stay away from buzz phrases and trends and think about it this way. There's a new set of tools that has a lot of weight behind them from the software development world that can be very useful for operating the network.
And I think it behooves me for my individual engineers to get software literate, and my organization to think about how should I operate differently because I have this new advanced tooling.
So I don't go out there and talk about net DevOps per se, but I I do say you you your network engineers should understand Python. They should know how to use Git. They should understand CICD for, configuration check-in. Like, they're not developing switcher router software or firewall software. Right?
But at those you know, there are common principles that I think are super useful, and the whole idea of being able to educate oneself and look at new tooling and technologies that's out there for network operations use, is totally within, you know, what Deming would have said seventy years ago if he had the same tools in front of him today.
Mhmm. Mhmm. Well, but tools aside, maybe that tools are not aside. They're they're critical. Right. We we are talking about emphasizing, processes, outcomes, the workflow that leads to those outcomes, and breaking that workflow into its parts yet at the same time understanding those parts in the context of the entire system, system, the entire workflow.
And then the cultural component involved with implementing those tools, using those tools, why are we using those tools, how do those tools affect these outcomes. Right? I mean, so that's really what we're talking about. So we're not we're not saying go learn Python as a network engineer, traditional network engineer, because that's the thing to do, because that's a DevOps technology. It's no. Because that's a a method, a technical, you know, tool in your tool belt to, to achieve those, workflow outcomes.
So those tools may change. It might not be Python in five years.
The words right out of my mouth.
Yeah. Yeah. But the but the concept would not. Correct. The underlying principle there. Right?
So so here's a way to glue this all together. Right?
There's a large service provider that I'm aware of that said we wanna go all in on automating our operations.
And what they did, like, to to prioritize, they needed to figure out where to start. Right? We've we've had this conversation multiple times. Right? What's the low hanging fruit versus what's the second tier, etcetera?
And they identified over a thousand workflows for things that happen within their organization.
And then they prioritized, you know, what should be done first. They found ways to assign costs to what happens with every workflow, especially when it goes, goes awry.
And they have a multi year plan for tackling automation of those workflows.
But workflow definition starts with what are people doing and how are they doing it, and how can we reengineer those workflows given new tooling that we have. It really starts with the process part, not not with Python, not with, Junos, you know, not with, you know, your latest new pluggable optics. Mhmm.
But given Deming's focus on leadership and the importance of that leadership in the organization, in this case, the IT or the network organization in particular, would you say that this is then largely a top down approach, and and the engineers are more or less the, you know, they're they're the mechanisms that carry out those the functioning of those tools and things like that?
I wanna be careful with, you know, broad sweeping statements here even though I maybe I haven't been earlier in this episode.
No. No. No. Don't be careful.
I love Yeah. I love broad sweeping statements.
Well, you know, different organizations function differently, and I do think there are ones that have rigor like this one, service provider that I called out. Like, they've decided to okay. If we're gonna do this right, we do need to start at the top. And and that workflow identification is not just a top down thing.
That means interviewing people who are doing the job. Mhmm. What what are the NetOps engineers doing? What are the NOC engineers doing, you know, at three AM for troubleshooting?
Understanding as is, you know, understanding present mode of operations so that you can optimize. Right? Okay. How do we make it better? Start with understanding the current state.
And I think I think that whole like, we we call workflow definition that has lots of things that hang under it.
I think many organizations have not had the discipline to actually do that. They've really just been figuring out as they go or it's the way we've always done it.
Yeah.
And I I really dig getting in and helping, companies think through that.
So, you know, that and five bucks will get you an extra large at Dunkin'. But, you know, being having to think through what are we trying to achieve here? How does it tie to business objectives and service level, agreements or service level objectives, you know, the new terminology?
Mhmm.
You know, those the connection of those things to workflow really matter.
And I and I I asked that question because, you know, it made me wonder if there's such an emphasis on leadership, how can there also be such an emphasis on you know, like, every every the buck stops here as far as the IT manager or the CIO. Right?
Right.
But then there's this emphasis on breaking down silos and open communication among departments and teams.
But I guess those two things can coexist. You can have strong leadership that also fosters, strong inter team, inter departmental communication.
Not only can you, but You should.
Okay.
Fair enough.
You must. And, like, I've had a couple really fun examples, in conversation and on the podcast of NetOps organizations that have brought in software developers.
And what does it look like to break down silos and think about how the different types of employee think.
And I had this one great conversation where, you know, the my guest, characterized it this way. You know, the the the ops the net ops folks were very sense of urgency. There's an outage. We need to resolve it quickly.
And the software developers were, yeah, eighty percent solution iterate. Eighty percent solution iterate. I've got time. I don't need to you know, there's no clock ticking on the wall.
You know, I'll fix what I can and check it in so I can leave at the end of the day. And that just very different view of, you know, their jobs and having to bridge the gap and get them to stop staring at each other from opposite sides of the room. You know, that takes that takes leadership facilitating those conversations, creating a safe environment for asking questions, all all that stuff. I in my time as a leader, I that's one of the things I've always enjoyed. Let's figure out how to do this together.
I'm I'm not smart enough to figure this out myself. There's a reason you're all here.
Yeah. And it's not to say that those therefore, the objectives should be identical, the software team and the network team.
Correct.
You know, when, you know, doing something eighty percent because you can pick it up again tomorrow might work on certain projects in the software development world. But you don't, like, say, oh, the core is eighty percent ripped and replaced. You know, we got eighty percent of the ACLs running. We're good. We'll come back on Monday and leave for the weekend. You don't do that with the network that moves every single packet.
Exactly.
You know, wireless is down right now, but, you know, we're working on it. And, you know, the hospital will be back online. All the wireless heart monitoring. They're probably not wireless, but my so my example is not good. We're we're definitely a more mission critical, more, we touch a wire, everything goes down kind of, you know, culture in networking. But the principle of still working together with these other teams, you know, in spite of those differences, I think is what we're getting at here.
Yep.
So that way, this system, this network, which does not exist on its own, it really is part of the entire system, which I I guess we can call the application delivery system or the service delivery system. The network is really just part of that. So when we're talking about, for example, observability and network observability in particular and and ingesting all these logs and stuff, I mean, we're looking at one part of this system. We're looking at the network.
And we like to say it's a system of systems, but it's actually a system of system of systems because the overall goal of the whole thing is to deliver that application to an end user. Or as, you know, on a podcast, like, a year and a half ago, Tony Efantis said to me, he goes, you know, it's really not even that. It's about delivering the data to the end user. Because when they log in to their bank account, it's not really the application they care about.
It's the data that they're they're receiving. So, you know, that's that's really our combined, effort. That is our joint mission, if you will, right, of the software team, of the developer team, and the network team, and the security team. You know, in in in spite of the fact that we're all managing different complex systems, it really is part of this one unified, goal.
Yep. Yep. You can't view it in isolation. You know, your your piece of the puzzle, has a ripple effect on everything else.
So you know what, Scott? On that note, I think we should end here.
There's a lot more to flesh out especially if we're talking about organizational behavior and people and that sort of thing.
So I wanna thank you for your allusion to Shakespeare and Julius Caesar in particular in that quote. And to answer your question, the, my favorite British author is William Blake.
So there's that. And We're good. I I would like to speak to you again one day about how we bring our past experiences, including education that might seem completely divergent from what we're doing today and how that applies to our careers today. Because I have some I have some thoughts, especially in in the context of AI and what's going on that I I think I'd like to explore with you one day.
So, on that note, thanks so much for joining, Scott. It it's been a pleasure. And to our audience, if you have an idea for an episode of Telemetry Now or you'd like to be a guest, I would love to hear from you. So you can reach out at telemetrynow@kentik.com.
So for now, thanks so much for listening. Bye bye.