Telemetry news now.
This is Telemetry News Now, a biweekly tech news podcast brought to you by Leon Adato, Justin Ryburn, and yours truly, Philip Gervasi.
So lots of news to talk about and so much AI stuff going on that I can barely catch my breath, but I guess that's why well, I mean, I guess that's the pace of our industry. Right? Or maybe I'm just getting old. I don't know. So let's get into this week's headlines. So in an article from Network World with the recent rounds of tech layoffs and general negative job sentiment, I guess you could say, in our industry, there is some good news for, I'm gonna say, most tech professionals for the upcoming decade.
So twelve of the fourteen computer related positions tracked by the Bureau of Labor Statistics. Is that right? BLS, I think that's what it stands for, are expected to see headcount increases over the next decade. So that's good.
But that's not all tech positions, unfortunately. The two categories that are expected to decline, being network and computer systems administrators, kind of broad categories, I think, which are forecast to decline by two point six percent. And, that's for computer programmers, which are also facing a decline of nine point six percent. That's decrease in jobs.
And specific to networking, the bureau is forecasting a net increase in jobs between twenty twenty three and twenty thirty three, so the next decade. And that's the, the three network related categories tallied together, and those are being at least the BLS. They kinda, describe those as being network architects, network support specialists, net admins slash sys admins. Again, all kind of broad categories. So a kind of a mixture of news there of good and bad, and, maybe some good for us working in in networking specifically.
Right. So I I think my take on this is, a, the industry is still growing, which if you've been watching actual job postings and stuff like that, which I do, and I actually collect them all up every week, and I I post them every week just to share with people.
There's always a lot of jobs, and they're not the fakey fake kind of job. I mean, these are real job postings from real companies that really are interviewing and hiring, maybe not as fast as some of us want when we're out of work, but they are hiring. So that's good.
I will admit to a certain level of skepticism about anything that says the number of net working jobs is growing. That doesn't seem to jibe with what my gut says.
However, I also have to presume that network here broadly includes, like, cloud networking and stuff like that, which which makes those numbers start to make a little bit more sense, just for my own internal, like, spidey sense.
And I think that's kind of something that we need to think about, that there are certain roles, certain jobs in tech that are slowly going to go away or slowly, change into something else that's, more in line with what we're doing today. So perhaps the role of the network engineer changes such that, number one, the job description changes, the title changes, and we just have to look at it differently. And I think I think that this bit of news, this headline sort of speaks to that a little bit.
Yeah. I think, you know, I'll start by pointing out that this article is from Networking World, so there's probably some bias in their reporting there. I think it's always important to consider the source when you're consuming news articles like this.
You know, that being said, it does support what I hear from the community that networking still they're still very much in demand.
Like Leon said, a lot of this probably includes cloud networking. And as we see a lot of more organizations adopting cloud for their for their compute for their infrastructure, you know, we're gonna see a lot more job roles posted out there for people who have networking skills and can come in and help those companies make sure that as they migrate those workloads to the public clouds that, they're doing it the right way, that it's being architected correctly, it's being secured correctly for for the networking portion of it at least. So, you know, it's exciting to see for those of us who are network professionals that there's still job postings out there and demand for the skills that we all have.
Yeah. I mean, when it comes down to it, you always need networking.
You can't get away from that just because of the nature of how we consume applications and various services today. It's always, you know, coming down to going over a network. So definitely some, interesting news there, some good, some bad.
Moving on, Blackstone to invest thirteen point three billion dollars in AI data center in the UK. Article also from Network World from a week or so ago. US private equity giant Blackstone will be investing thirteen point three billion dollars, which is about ten billion pounds, to build, I believe, one of Europe's largest AI data centers in Northumberland, UK. And this is from a statement from the British prime minister's office, by the way.
So the project is expected to enhance the UK's AI infrastructure, whatever that means, catering to the vast data storage demands of AI technologies. And that's interesting to me. It does seem to be focused on the storage, component there. Now the prime minister's office also said, and I quote now, the deal with US investment company Blackstone facilitated by the Office for Investment will create the biggest AI data center in Europe, boosting the UK's world leading capabilities in the AI sector and driving growth in the local community.
In other news, governments like to use very long sentences in their formal statements. Anyway, construction is planned to start sometime next year.
Yeah.
I mean, I think you hit on it a little bit, but the one thing that I found odd in this article was that it really focused on the storage and and the data itself.
I think the quote I took out of there was catering to the vast data storage demands of AI technologies.
We all know that AI needs large volumes of data to train the models and do the inference, but, you know, usually, we see the bigger investment being made in things like power space cooling, the GPU nodes themselves, switching and cabling that connects all those terminated GPUs together to process all that data. And there really wasn't much talk about that in in this article, which I thought was was kinda interesting. It it, at least, I took away from the article that the ten billion pounds that's being invested to build this is mostly mostly focused on the building itself and and the storage.
So it'd be interesting to see, like, what are they doing for, power? As we all know from some of the shows we've had over on the Bullet Tree Now podcast. That's a big constraint when it comes to building out these, data centers is the how many megawatts is each data center gonna be able to produce and so forth. So I think that would be the Mhmm.
Interesting thing to know about this, data center. What kinda what kinda power is it, gonna be consuming, and how are they gonna provide that power?
So I I wanna jump in and, first of all, admit that when I started reading the article, I saw Blackstone, and I really thought my brain just switched it and said Treadstone, which is the evil project in the center of the Jason Bourne movie and book series.
So I have this completely unnecessary bias still lurking in the back of my mind. With that said, according to the Blackstone website, not the Treadstone website, the Blackstone website, Blackstone is, quote, a leading global alternative asset manager, unquote.
And just a little bit of Internet searching I did, it is the largest commercial landlord in history, according to the register.
And, it's involved in everything from Spanx to Bumble to Legoland, which almost covers the entire dating cycle, if you think about it from a particular point of view.
That's true. Yeah.
But that's what, for me, makes the article even more spicy. Like, why is Blackstone getting involved in something like that?
I also find the whole thing interesting because today, the day that we are recording this, is the day that the UK shut down its last coal fueled power plant in the country.
So just knowing the enormous strain that AI puts on the power grid, Justin, you were just talking about that, I wonder I wonder what the impact will be of spinning up an AI data center. And is is everything in the UK prepared to handle a project like this? Of course it is. The question is how.
I don't know. It just all struck me, but I'm just a little leery about Blackstone, the alternative asset manager investment company, getting involved in AI.
And I I don't know. It just I mean, you know me and AI. I'm a skeptic, you know, all the way through, but still.
Yeah. I mean, this is more likely a real estate investment than it is in an AI investment, considering that Blackstone is a private equity firm interested in high value real estate. And there's some speculation going on here, of course.
I don't I don't think that they're necessarily interested in the technology of AI, nor do I think that they're necessarily interested in the long term investment of, like, like, generational investment and, you know, the benefit to humanity or or even from a a pure profit standpoint. From a in a real estate investment standpoint, there is, I mean, you go to Northern Virginia, it's all data centers.
You think about other businesses that, you know, on on the surface look like a a restaurant or whatever, but underneath, they're really real estate investment firms. But that's my guess there. But, you know, the the the language of the article did speak to storage and, quite a bit, which is interesting because many of the models, when I say models, I mean, foundational models. I mean, there's almost a million models out there, large language models in particular since that's what a lot of people are really talking about when they talk about AI today. Although it's not the only type of AI, but that's usually what it is.
But the foundational models, the big ones that we're familiar with, like GPT and Claude and and to a lesser extent, the ones that, like, Meta puts out there with Llama and stuff.
They, you know, they're often trained on the entire Internet worth of data, right, like GPT is known for. So what what specific storage are we looking for? What are we looking to store? What kind of data is this?
Is this part of, some other system that we're ingesting telemetry from maybe IoT devices around the UK and ingesting it so that way we can analyze it and do I mean, I I don't know what the actual goal here is. That's not part of the, the text. It's not necessarily something that we have purview over, so I don't think we're gonna know. But it does speak to also the you know how, like, we had the arms race in the eighties?
I mean, is this, like, the AI race now?
I don't know.
Hundred percent. Hundred percent. It this is there there have been so many escalations between, you know, data storage race and there's you know, there was a race for for bandwidth speed for a while. So, like, yeah.
Yeah. So in this case, I really think that this is Blackstone recognizing that, you know, data centers and specifically AI data centers are going to be growing in numbers such that owning the real estate and being partnered with them from that perspective is going to be incredibly profitable. Again, not necessarily an AI endeavor, but profitable as far as a real estate investment.
And with that, we can move on to our next headline related from The New York Times behind OpenAI's audacious plan to make AI flow like electricity. Now this is gonna this is a pretty long article, so I'll summarize it relatively briefly here for you.
It really needs to be understood all as well alongside other related news. But, basically, Sam Altman, the CEO of OpenAI, has initiated a large scale plan to secure the computing power for future AI projects.
And what he's doing is proposing the construction of data centers and chip factories, and this is worldwide, including in the Middle East. So initially, it was kind of floating around out there that he was looking for trillions. That's trillions with a t r I l l I o n s of dollars. But, of course, he faced some pushback from US officials over foreign infrastructure. Right? And so that got scaled down to a measly hundreds of billions of dollars.
And, I mean, really, Altman's vision involves creating AI infrastructure, right, akin to electricity distribution. Think about how that works. And so, like, kinda pretty much with the goal of making AI tech very, very ubiquitous, very common, and, as the article suggests, transformative to society as much as electric power, electricity was.
So, you know, Altman does seem focused on building data centers in the US.
He's been courting government officials and investors like Microsoft and NVIDIA.
There is so much to unpack here. So many ideas around the commoditization of AI. Is it does it become a utility? Do we have foundational models that just become kind of in the background that we all consume?
How does this stuff get monetized? How are these companies really gonna make money? What is the role of AI in society and culture? There's so much stuff.
And, you know, we were just chatting in the previous article about, you know, the AI race.
How does security fit into that? And, you know, balancing government and private investment. I don't know the answers to all those things, but really, really interesting stuff.
I'm gonna jump off for a second and just say that that if you're gonna compare anything to electricity, please remember that one of the first things they had to do before running into people's homes was make it safe.
Because, you know, in the lab, you know, fine. Everyone knows not to step on that power line or not to touch these two things together or whatever. But if you're gonna run into somebody's home, you're gonna make it ubiquitous.
There's a lot that happens. And as much as we think that electricity, you know, happened overnight, it didn't. It took decades to to go. And and I just you know, I I need to see a lot more safety involved with with just basic AI before and and on and and usefulness.
The other thing about electricity was, what did it do? It ran devices that would have required significant effort for humans to do, whether we're talking about, oh, you know, washing machine, a mangler, you know, the the pre dryer, the light bulbs, like stuff like that. Yeah. You know, as as many people are fond of saying, I want AI to cut my lawn and fix my plumbing.
And while so that I have time to do art, not the other way around. And I don't see that happening right now. So I don't know.
There's I don't know. Some parts of this article jumped out at me like sentences from a spy novel, You know? In meetings with investors, United Arab Emirates, computer chip makers, and Asian officials in Washington, the billionaire c v CEO proposes they unite in a multi trillion dollar effort to erect new computer chip factories and data. No. This is, like, right out of a nineteen eighties apocalyptic novel. Like, no.
Yeah. And and it does make me think about how we've been talking about the Internet in the past generation, you know, The access to the Internet. Well, I mean, to the information that we can get via the Internet, so it's not like the Internet itself. But, you know, for all intents and purposes, the Internet as, like, a human right, whether I don't know.
Whether that's true or not, that is that is a discussion that's being had right now and has been for quite a while. So is AI kind of the same thing where access to this very powerful technology is, like access to municipal resources like water and sewage and electricity. I don't know. It's something to think about, for sure.
Yeah.
I mean, I think my favorite line in this article is, it was also a demonstration of the tech industry's determination to accelerate the development of a technology it claims could be as transformative as the industrial revolution. That's something I've been saying for a while now. I think this is gonna be our next industrial revolution or the next invention of the Internet as far as its impact on our society. Now that is on a long term, long time horizon kind of thing. But in in all seriousness, I think this makes total sense for Altman. Right? I mean, he he he's running OpenAI, which is one of the largest developers of these large language models.
In order for his company to be successful, there has to be, chips that are out in the ecosystem. There have to be data centers, like we were talking about in the last article, that are built for these environments. There are a whole bunch of, like, ecosystem around his software and his model development that has to exist in order for his company to grow and scale. So, I mean, it makes total sense that he'd be wanting to invest in some of these chip design and chip fab, type of things, which, by the way, reminds me of a great book that I read, recently that I think our audience might really like, which was called Chip War. It's written by Chris Miller.
It it documents the history of, chip device chip design and the manufacturing, of some of those and the the geopolitical issues that go into that and are and are behind that. So I think it's I found it really fascinating. It was really entertaining writing style, but just some great history on Philip and the manufacturing of silicon that, is gonna be ever more present as we see more of these GPUs, more of the AI processing being done. We could put a link to that in the show notes for anyone interested in checking out that book.
Yeah. Yeah. Thanks for that. That's great. You know, this is an interesting thing because, well, it's not just OpenAI.
So what he's doing to sort of evangelize how pervasive AI is going to be, at least in his opinion, and then, locate and allocate the resources to make that happen. I mean, there's other there's other foundational models out there. But, also, there are many folks that are sort of rethinking this saying, what about, like, small language models? Instead of LLM, you have an SML or SLM.
Sorry. Excuse me.
So instead of five hundred billion or a trillion parameters, I have a seven billion or a forty billion parameter model, which is still a gigantic number, but it is it pales in comparison to the very large language models. And they're trained on very specific data, which reduces, among other things, the incidence of hallucinations. It makes them contextually specific and relevant, easier to run, cheaper to train and operate. So that's a thing that's happening.
But I think one thing that both of you guys said or at least talked around, in this conversation so far is I don't know, and I think I agree with you in reading between the lines here. I don't know if anybody really knows the long term value here. And it's it's hard to say because as a tech person, this is cool stuff to me. I am I'm playing with a lot of this technology at home.
I'm, doing a lot of research. Really cool. But cool doesn't mean, like, valuable to my business.
So was o s two warp. Yeah.
We could talk about, like, the apparent immediate benefits that we're starting to see from the application of LLMs. Really, it's large language models and a greater AI workflow. Two, IT operations or operations. You know, we talk about the reduction of mean time to resolution. We talk about the, how much easier and faster it is to interrogate huge datasets and derive meaning from very disparate data as well, which normally takes, you know, a lot of effort, and and therefore isn't really even done.
And so I think, you know, all that's true, but the entire industry as a whole really is trying to figure out, okay. What is the what is the real benefit to my particular niche of tech or in the broader scope of things in my particular industry? Because we do know that there are applications in, you know, in health care, in aerospace, in, you know, other industries.
So in fact, that reminds me of a conversation I had with my buddy the other night, and this is just a personal conversation with my friend who is, he's at a national health care organization. He's, like, a senior engineer, technical role. And he was telling me about how, the leadership, you know, is telling him for twenty twenty five, their goal is is AI. You know? How we gotta use AI. We gotta incorporate AI, and that's about all they're saying. They don't really know exactly why or how.
All my friend knows is that he needs to use it, incorporate it, and he's sitting there like, okay. Yeah. Some of this stuff is pretty neat, but, like, I don't see how it's gonna provide value yet. Or or or is it gonna just disrupt my entire life and work flow?
I mean, we have a bunch of senior engineers that do stuff. You know? He's telling me, in fact, how he, like, has this system with, like, Chrome runbooks, and and it, like, solves so many of their problems autonomously. It's really neat.
And so and so what is the value? Right now, we have a perceived value, and we have some inkling of where it could go, but I think the industry is still trying to figure this out big time.
Yeah. I think I think that's where our industry sorry, man. I think that's where our industry gets into trouble a lot of times as we build solutions in search of a problem instead of the other way around. Right?
If you really think about the comment that your friend Paul made, what he's saying is, like, we're gonna do AI whether it actually solves a problem for us or not, which doesn't make any sense. Now that being said, sometimes you don't know what technology is going to be capable of or how you can apply it until you spend a little time playing around with it. Right? And so I think a lot of companies are experimenting with AI and saying, okay, is this gonna solve problems for us or is it just gonna be a, a trend or a fad?
Right? And so some of it requires some research in advance before you figure out whether it actually solves the problems that that you have in your business or not. So that's I think that Yeah. The truth is somewhere in between.
Do you guys think that this really is a technology that's gonna be as transport as the industrial revolution was itself? I think it has the potential to.
I I to put things in medical terms, I I understand and believe in off label usage.
You know, that's fine. And and sometimes the off label use usage becomes the primary usage or the more important or the more impactful usage. But I don't know. I think right now, we're still so much in vaporware market texture, hype cycle garbage that it's I have a hard time separating it.
And, again, too many people are focusing on the sizzle and not enough on the real state. Like, again, help me cut my lawn and fix my plumbing. Mhmm. Do not help me write my blog.
Yeah. Yeah. And and in related news, OpenAI is recently undergoing a pretty significant transition. Right?
From it was originally a nonprofit structure to a for profit entity and, you know, to become a you know, it it was this idea of it was gonna be a public benefit corporation. And so that shift, Justin and I were chatting about it, earlier is probably to attract more investors. Right? And, Sam Altman is certainly gonna be much more wealthy as a result.
But there is, some concerns that have been raised. How is OpenAI gonna balance this mission of developing AI for this huge transformative thing? Whether it is or it isn't. Right?
That's the idea.
Balance that with profit driven motivation. Interesting stuff.
Okay. So, I'm gonna take the next one, and this is where I probably say, and now for something completely different, at least different from the conversation we were just having. The headline is man charged with selling forged license keys for network switches.
And the summary is that the US government has indicted a man for his participation in an international conspiracy to sell forged license key license keys for network devices.
Benjamin Paley, co owner of the IT service company Gen eight, and his coconspirators have been indicted for international conspiracy to traffic counterfeit net network device licenses. The scheme ran from twenty fourteen to twenty twenty two, and it specifically involved the counterfeit license keys for brocade switches.
That's the the summary. I'm gonna kick things off just by saying, who among us hasn't worked for a company that had exactly one of each type of Cisco device under contract, and every time you had to open up a TAC support case for any of them, you use the same serial number.
Mhmm.
Is that just me?
Not just you. Maybe. Just you. A little different from what this is go what was going on here, though.
Yes. Little different. But, yeah, I get what you're saying, though, as far as being a little sneaky there. Yeah.
For sure.
So the company I'm thinking of where I where this was a a massive widespread practice also felt no problem whatsoever with using, you know, cracker programs to generate big license keys for the software that they were running in house.
They had no problem that Yeah.
I I'm not so they had no problem running massive numbers of extra Microsoft licenses and not paying for it. Like, it was it was a different time.
It it may be worth saying that, mister Paley is seventy five, so he may be operating under a different set of IT assumptions.
I'm I'm gonna weigh in sort of ethically and say that while this is not new and while many of us have been in companies that do this, it's it's not helpful, nor is it particularly profitable. That's the point I wanna underscore here is that every place I've been that does this never ends up saving a ton of money or making a ton of money off of it. It catches up to each organization that I've seen this ad. It catches up ridiculously quickly, and the penalties far outweigh the, you know, any perceived benefits.
It's just dumb. Yeah. I'm not surprised that somebody's been doing this. I maybe I'm surprised that they got away with it for so long, but it's I would love to see their balance book and see what actual dollars they made because I'm betting it's not a lot.
Yeah. I mean, I, you know, used to get frustrated when I worked for a hardware manufacturer and customers would buy used gray market gear, buy optic from somebody else, buy, you know, licenses, hacked licenses like Leon was just describing, especially when you're the staff putting in long hours vetting out the solution, working with the customers, and then they go buy the gear from somebody else. Right? But I do get the fact that, hardware vendors charge us King's ransom for some of these newer gears and some of these licenses and especially with software based licenses.
Like, you know, how much are they actually how much investment do they actually have and what they're charging for some of those licenses. But, you know, there is an ethics piece of this too to it too, right, which is, you know, you're you're basically akin to stealing something from the vendor. Right? So, you know, I just think if you're working for an organization that's that's budget constraint, I can understand buying through buying stuff on the gray market, but just gotta be really careful that you're still buying it through the authorized reseller and you actually have authority and, that it's actually the legitimate thing that you're purchasing and not something that's, you know, ransomware or something that's been stolen that that you're putting into your production environment.
Because that can have real business risk for you.
Yeah. Business risk as in, like, lose your business and go to jail. And you gotta weigh that out. You know?
Is it worth it to go down that road? And you do the crime. You do the time. So moving on, just crossed my desk yesterday, a new Cisco vulnerability.
Cisco catalyst ninety or nine thousand series switches denial of service vulnerability. This is from their Cisco documentation published published on September thirty.
A, vulnerability in Cisco IOS XE software could allow an unauthenticated adjacent attacker to cause a denial of service condition on the control plane of an affected device.
We are talking about nine thousand series switches as far as the affected devices. This is due to improper handling of frames with VLAN tag information, which means an attacker could feasibly send custom crafted frames to an affected device, a network device, which would allow the attacker to then render the control plane of the affected device unresponsive. That's a very bad news if you're a, a network device or managing network devices, more likely.
So the device would not, not only be, inaccessible through the console or CLI, it would also not respond to ping requests, SNMP requests if you're still using SNMP, which you probably are, or requests from other control plane protocols. And, and the only recourse we have is a reload of the device to restore control plane services. This is bad news. So so far, Cisco has released software updates that address the vulnerability. That's good, but there are no other workarounds.
And as of now, this affects Catalyst ninety three hundred x switches, ninety four hundred x supervisor engines, ninety five hundred switches, and, the beautiful chassis switch, the ninety six hundred switch, all running iOS XE.
So, that is for your information. If you are running any of those platforms, you should, take a look into that immediately.
Now moving right on to the next headline, Meta fined one hundred. Meta, as in the company, Meta, Facebook fined one hundred one point five million dollars for a twenty nineteen breach that exposed hundreds of millions of Facebook passwords. Take it from TechCrunch from last week. So Meta has been fined a hundred and one five point five million dollars by the US government specifically for a twenty nineteen data breach that exposed hundreds of millions of Facebook passwords. Now the problem here is that Facebook was storing users' passwords in plain text, and that, of course, makes them vulnerable to access by employees, other unauthorized parties that are integrated with Facebook.
And here, the fine represents a a pretty significant penalty for Meta, obviously, over a hundred million dollars, which has been faced with, ongoing scrutiny scrutiny, excuse me, over the past few years over privacy, data security practices.
And, you know, despite the breach being disclosed back in twenty nineteen, it's taken this long. You know, regulatory actions, penalties, they continue to unfold five years later, highlighting how serious this is, especially when you're thinking about privacy issues.
I I love the fact that Europe is embracing and leaning into its role as the tired, incredulous, cranky, slightly misanthropic preschool teacher who is forced to put companies in financial time out every time. And I just wanna know, does GDPR guidelines include provision for public flogging of executives? Because I really think it should be.
Pub public embarrassment, at least it appears.
As yet, I do not believe that public flogging has been written into the letter of the law. So no. No. I don't think so.
I I don't know. When you have no shame, how can you be embarrassed? I mean, like, how many times I've been off Facebook since twenty fourteen.
And I and at that point, the number of breaches had become egregious to me, and it's only gotten worse since. So how how much more?
Yeah. I think this does speak to how there is more being, more folks from the very highest levels of government included here in the United States paying attention to, security concerns, breaches, inappropriate use of personal information, from customers as as something to pay attention to. Whereas prior, you know, I remember years ago, it was still a debate. Like, this is a private company.
You know, you have a relationship with them. You're giving them your data. The government has no role in that. What if there is a breach?
You know? And so people talked about it in the context of banks and and credit card companies, but now it's expanding. Here, we're talking about social media. Now to be fair, a hundred and one million dollars is not that's petty cash.
That's like, you know, what the admin assistant has in the safe, you know, behind the desk at at the headquarters at Meta.
Zuckerberg probably spent that on dinner last night. Yeah.
Yeah. Yeah. Exactly. That's just like what they have for coffee, petty cash to cover, like, staplers and things.
But it is still significant. It's a lot of money. Sure. But to me, it does speak to, more and more attention being paid to this and, the government taking this seriously and not as, something that's just between consumers and the companies that they're in a relationship with.
So, where that's gonna go, I don't know, because this is a small amount. This could you could almost, put this as a line item in your budget if you're a company this size, or you could say, screw it. I'm never gonna encrypt anything, and I'll just pay the hundred million dollars a year. I don't care.
It's cheaper to do that. That was an old Greg Farrow thing, from years ago. I heard him say it in networking field day, so I can't take credit for that.
Yeah. And and I think that that Europe and GDPR is gonna keep stepping up and saying, no. No. No. No. It's per occurrence.
Mhmm. Yep.
It's per occurrence. So, you know, you four hundred million passwords.
Okay. Let's go. Yeah.
So I don't know. I just it frustrates me and no one.
Alright. In other news, Verizon says resolved network disruption, FCC probing company's mobile network outage for thousands of users in the US.
The summary here is that, as many people are probably painfully aware by now, yesterday as we're recording this on September thirtieth twenty twenty four, Verizon experienced a widespread network outage affecting over a hundred thousand users across the United States.
They should begin in the morning with outage reports coming in from Chicago, Phoenix, Omaha, and Washington DC area users.
Many of these users found their phones stuck in SOS mode, meaning they could only make emergency calls. Verizon engineers worked throughout the day to resolve the issue, and by the evening, the company announced that services were largely restored.
The, FCC, of course, is launching an investigation into the cause and scope of the outage.
And although Verizon has publicly apologized for the inconvenience, they still haven't provided specific details on what caused the outage.
I will say, personally, my suspect my suspicion here is that our RFO is gonna come back, reason for outage, by the way, RFO is gonna come back as an eSIM provisioning issue if you think about, the symptom there that some phones were showing us SOS while others weren't. I know personally I had some family members who were on Verizon living in the same house, presumably connected to the same cell phone tower, and one phone was working and one wasn't one was showing that SOS issue. That to me smells like, some sort of eSIM provisioning issue, but I guess we'll have to wait and see what the what the RFO comes back as. Okay.
I was seeing comments about eSIM for, also, there there were a lot of outages yesterday. I I saw AT and T, Verizon, PlayStation, the entire state of North Carolina.
I think at one point, some were weather related. There there was a for those people who don't remember, by the time you're listening to this, there was a hurricane that knocked out a large part of the state. So, I wanna personally avoid the Tuesday morning quarterbacking and focus on some of the lessons that I we can take away from this. First, if you're in any way associated with an enterprise network of any kind, backup circuits on a second carrier are still a good choice even if you go old school with, like, you know, the old cell phone WIC cards on a router. It's just something to fall back on when the first method doesn't work. And and second, and I know this sounds self serving because of who sponsors this and, you know, who pays my mortgage, but monitoring and automation are are and always shall be your friend.
Network outages are not complete mysteries. They are what we call a known unknown. We know it's going to go down. We just don't know when.
So don't walk around pretending the face eating leopard isn't going to eat your face. Like, you know, and the face eating leopard in this case is called the backhoe operator who dug up my fiber line or someone else's fiber line next door. Just, you know, have monitoring that tells you when things happen so that you can react appropriately and quickly, you know, with the right methods.
Yeah. Yeah. And, I mean, Verizon has publicly apologized.
I think they use the term inconvenience, right, or disruption. Nevertheless, the FCC is launching an investigation.
So backup circuits or not, Verizon is looks like it's gonna be on the hook to explain themselves, which, you know, any reputable organization should do.
And this also comes on the heel of, the heels of some really big, you know, expansions in their footprint, and we've discussed some of these things. So, yeah. Yeah. This is, this is an important one. I mean, when we rely on these carriers for mission critical activity, sometimes, you know, mission critical in the sense of, like, lives, people's lives. So, the FCC is launching an investigation and stepping in, into the the cause to the to the actual scope, in order to hopefully remediate that and and and not allow that to happen again, I think it's important.
So Yeah.
I mean, I like to say there are two types of organizations, those who have had an outage and those who are going to. Right? So we're not gonna benefit much from trying to poke at Verizon for this particular outage, but I think Leon's got a really good point. Like, what can we learn from this?
What can we take away from that? Right? One is definitely redundancy. Right? Especially in twenty twenty four with five g being as ubiquitous, at least in the United States as it is, you probably get a backup five g circuit for, like, fifty bucks a month.
I mean, that's not a huge investment in your overall IT budget to be able to have a backup circuit, you know, especially for something that's critical to most, most organizations as being connected to the Internet. I know personally in my own home when the Internet's down, I'm probably the last one to know. And then, you know, when I got into this industry, I was the only one who cared. The Internet went down.
Like, our our reliance on the Internet has become that critical. Right? And same thing in enterprise organizations. You know?
If you're you got a branch that's completely offline and no one could connect anything, they might as well go home for most of your workers.
So Yep.
For sure. Alright. So, rounding out today's episode, let's go over upcoming events.
So first off, on October third, that is the day that this, episode drops, actually. The New York networking user group, New York Nug, in Saratoga Springs, New York. That's the event that I personally lead along with Zig Zsiga, and, that is part of the USNUA, but that will be this, this Thursday, the day again that this episode goes live, October third. If you're in the upstate New York area, capital region in particular, come on out. Free pizza, free beer, free chicken wings, and I'll be doing a presentation on submarine telecommunication cables.
On October ninth and tenth, we have dev ops DevOps Days Dallas.
The Florida network user group, I think I don't know if this is the first one, but, there aren't that many in Florida. So the Florida Nug in Tampa will be on October seventeenth. I'm not gonna be going to that one. I'd like to. Security field day, October twelve I'm sorry. Security field day twelve is October sixteen and seventeen.
Special shout out to, Justin and Doug Madory, our own Justin Ryburn and Doug Madory, who'll be speaking at NANOG in Toronto on October twenty one at three twenty three. Justin, what are you and Doug gonna be speaking about?
Oh, you're gonna put me on the spot a little bit here. I'm not actually sure what the topic of Doug's talk is. Mine specifically is on BGP flowspec, which is one of my, nerdy passions as a technology that hopefully can help us, mitigate DDoS attacks.
So Okay. Great. Pennsylvania Nug, the PA Nug, that'll be in the Philly area October twenty fourth. I went to that one, over the summer.
Actually, took my daughter that. It was it was great. It was really a lot of fun. So I may be going I'm not speaking there, but I may just take the road trip.
It's four hours from me, so why not? The next Ohio nug in the Cleveland area is November seventh. And, Leon, I believe that's your neck of the woods.
It is absolutely my backyard. I will you know, god willing in the creek don't rise, I will be there.
Great. Networking field day thirty six. I can't believe we're at thirty six already. My goodness.
Is, November six and seven. So make sure to catch the live stream for that. And last but not least, on this particular list, we have AutoCon 2, November 20 through 22. Now, if you have not purchased, your tickets for that conference, they are still on sale, but they are going fast.
I I think the workshops are filling up pretty quick too. So go to network automation dot forum. Check that out. Sign up for what you want, or just go to the general conference only.
But you wanna you wanna get that squared away, especially because the hotel may be full.
Now done with those other events, Leon, I'll pass it to you.
Yeah. So, since we're looking at the entire month of October, for folks who are observing Rosh Hashanah, Yom Kippur, and or Sukkot, over the next three weeks, Everyone at Kentik wishes you a Chag Sameach.
Great. Thank you, Leon. So thanks so much for listening to the news today.
I'm Leon Adato.
I'm Justin Ryburn.
And I'm Phil Gervasi.