Internet traffic and current events with Doug Madory

Hi, and welcome to Network A. Today I'm talking Doug Madory, the man who can see the internet. We're talking about internet and the news. What's going on in the world and how it's affecting and being affected by the internet.

And talking about Ukraine, Egypt, and outages there, and and other effects, and internet censorship, seems to keep happening for various reasons. If you're curious at all, please give it a listen. If you're interested in network a f, you can find us on podcasts and, favorite us there. We will link the, show notes today to this episode and also some of Doug's blogs.

You can find him, Doug Madore, a blog on the Kentic blog, and also Doug Maduri on Twitter, d o u g, m a d o r y.

Hi, and welcome to network a f. Today, we're doing a current events update, and I've got with me, Doug Madorey, who's been on network a f before. Doug, you wanna introduce yourself quickly.

Hey, Abby. Yeah, I'm, Doug Madore, I'm the director of internet analysis for Kentech.

And where are you physically right now? Presently in Bond, Germany, I was invited to be a panelist at the, Deutsche Villa, which is the, public broadcaster of Germany. They have an annual conference called the global media forum, and, I'm in here talking about internet disruptions.

Cool. And I guess we'll talk about some of that today.

For those that don't know, you are also known as demand that that sees the internet, and have done that at, multiple companies and most recently at Kentech, where we enticed you, with visibility, not just BGP, but of traffic performance and and other data.

I think the the thing really topical, over the last months, and top of mind for everybody is is Ukraine and what's going on there.

Obviously, there's a lot of different layers of that, but underneath keeping people connected is the internet.

And I guess, I'll just start off what's going on with internet infrastructure in Ukraine? What's been happening recently? Yeah. We haven't spoken on this, topic on this podcast about Ukraine, but, you know, the invasion took place on February twenty fourth, this year, twenty twenty two. And, for the next month or so following that, that was a pretty busy time for me just trying to keep up with what was going on. I think a lot of people were keeping me up what was going on in the news. And, and there was a lot of things happening internet wise.

And now the, the conflicts which you know, sees no sign of resolving anytime soon has evolved into a few different I don't know what stage we're in now. It's definitely not the we've left the first stage or maybe we're on the third stage or something where the It's a different, horrible stage. So Yeah. So the you know, in the in the out in the outset Ukraine was getting attacked from every direction.

We were seeing all kinds of outages all over the country.

Now the the impact seems to be focused mostly on the east, and the south of the country.

And that's in keeping with, you know, if you follow the news, you know, the the Russian strategy now has been to try to focus its efforts on those areas that they can play away from from Ukraine.

And I guess the most recent development is just seeing some of, the, the network size piece to operate in Ukraine, getting, plucked off and rerouted, through, through Russian transit.

So in the city ofkerson in the south, at the beginning of May, there was a there was an outage. The city was down for a couple of days, and then one of their, name providers came back up using Russian transit, basically going down south through Crimea, through Miranda media is the Ross telecom rust telecom is a state telecom of Russia. They have a local agent in Crimea that they set up after they annexed Crimea in twenty fourteen.

And so this Miranda media that became a transfer provider briefly forkersonde Telecom, which goes by another retail name, and I don't recall Sky something.

That only lasted a couple days. They were reverted back to Ukrainian transit, and the CEO published something on social media saying I I had no choice. I had to connect this, to keep us online. And then that story seemed like it was over, but then towards the end of May, then they there was another outage that they switched over to, Miranda Media again. And they were the only ones. There was, you know, it's been about six or seven, ASes that we've seen, switch over to Miranda media in that region.

And it's, it's not one single event. It's, some of these are, on different days that are getting switched over.

So the implications are for anybody who's living in those areas that uses service, there that, you know, your, your internet and telephone, service could be surveilled, intercepted manipulated by, the Russian government is the risk these folks faced face now.

Yeah, and that's just that's the latest. And so, for the most part, it seems to be people still in control of the networks in Ukraine, but being forced by economic or other, you know, projectile weapon based incentives to, form the right transit relationships so that the packets go through unknown infrastructure that could do whatever to them. Yeah. The, I mean, I I heard from one of our, Ukraine contacts, one of these was a a shift at gunpoint.

And so these are the same engineers that keep the telecom up but they are now, forced to to route through, crimea. Is that, first? So, you know, a hijacked as in, you know, a route jack, as far as you know, oh, you know, not via BGP exploitation, but by we'll say what Yeah. I bet that's happened before.

I mean, I I can't think of an example. It comes to mind, but, I doubt I doubt that's the first, first time that that's happened. But I I would say this that the, the second time around when providers started switching over and cursed on up to Miranda media. It wasn't like the first one where there was, like, completely outage and they had to bring things up.

One of the providers switched immediately. Like, it was it was the circuit was ready already ready to go. And so, maybe this was a bit more of a planned, switch over, than what had happened to the beginning.

Now I guess our PKI wouldn't be affected because it's only origin validation.

Right. That's not not useful in this. Do you have any, you know, any any of their transit providers use route registries or required updates that could that could cause a partial outage, you know, you know, the first time. But, I mean, it's pretty it seems like it's definitely a little bit of a shift forward in a we'll just say, to be non judgmental, we'll say expands the universe of traffic engineering methods, but, you know, definitely, not good.

Now it's it's interesting because early on, it's a different kind of networking, maybe Russia hadn't thought about the implications of using roaming cell phones, for soldiers and had some, you know, implications there. And early on, it looked like, you know, the internet was really being as reliable as you would expect it to be as long as one last router of whatever type it was was, like, up and connected. Like, It was really only when the physical damage got, you know, almost complete that those networks, you know, were going down. And even in those cases, you know, like, it's been, I think it public publicized and, properly, rightfully celebrated.

There's been a lot of, technicians in Ukraine that add great risk to themselves have gone out in fresh rubble to go, reconnect fiber optic cables destroyed in in in the fighting. So that's, you know, I take pride. This is our industry. You know, you and I, we're all in this.

Those are our, those are our people. And that's, it's, After nine eleven, there was, I should really remember the gentleman's name, but he, basically, he was posting from one of the mailing, Naddogg, I think. He was in telehouse, which was in the, you know, no go zone.

Twenty five broad, broader Broadway. I should know that. Down in broad, I think.

No, Broadway, down in in in lower Manhattan, and he was doing hands and eyes for basically everybody because he was in the building. Okay.

You know, that was maybe uncomfortable, but less dangerous.

Yeah, people stepped up to, to the challenge.

That's what's happening, Ukraine.

So if you look at the total amount of traffic to the country relatively, is is the man who sees the internet.

For the most part, there's still a tremendous amount of internet activity and you know, was it is it down? Yeah. There's been a decline since, February twenty fourth by volume. We're just looking at bits per second.

No. Not even that. Not even that, I mean, maybe twenty percent or thirty percent. You know, it's a it's a fluctuating, signal.

So, But I it's it's not I wouldn't say it's half. And so there's a there's factors there. There's, you know, there's been millions of people who flood the country. So that's those are those are internet users that are no longer, using the internet that are gone.

There's been things that have been broken And that obviously is gonna hinder, internet use. And then also that, for the people who have stayed life is not quite as it was.

So they may not be netflixing as chill and chilling as much as they, did prior to the twenty fourth of February so that can affect, traffic volumes. So there's been a, you know, noticeable decline, and we've seen this. I I think I posted a a list of what we saw, a bunch of other, outfits in public, public, yeah, cloudflare, Google, the Iota, which is completely different, you know, set of data and it now actually even tracks. So the Iota data is the internet outage detection What's the eight stand for? Project, that's out at Wood Georgia Tech now.

And, and so this is, you know, the the b to b routes the active measurements, so pinging things. They have this litter thing where they measure background traffic coming out of, Ukrainian address space and all those those are different metrics than traffic volumes, and those two have dropped, you know, with some, not to half, but, maybe twenty five percent, thirty percent.

But otherwise, the country stayed online. It continues to stay online, and I I don't foresee it.

You know, the country being disconnected.

Well, hopefully, things go as well as they can be can possibly go for the people there, as well as for the infrastructure.

And, Hopefully, how to VPN does not get blocked on, on search for people from those affected networks, but my guess is there's already some VPN going on there. I know there are some special offers that that, you know, we're going out basically just to help connect people, people reducing, you know, or not charging for access, you know, to different services, you know, for people from Ukraine. So in other excitement, Egypt.

There's all sorts of jokes about sharks, you know, with lasers cutting, you know, in international. Yeah.

But this was not on, this is not subsea is where cables that were affected, on land.

Yeah. So it's interesting situations. This this occurred, while we were at Nanogg, up in Montreal and, saw the the traffic, fluctuation in a bunch of countries confirm that that the this was, Subsea cable related, and and I used the term. If you'll look at anything I put out, that there was a an outage meaning that service was on a, but I, but I didn't use the word cut because I don't know that the cable was cut because that incident, in event.

That means something else. Alright. You you can stay out cause the service was not working on the cable. So it was out, but not necessarily cut.

You have to be careful because of cut.

And I and I, in in reality, I, you know, the the summary cable, world is so opaque.

It's very hard to get, you know, a full accounting of what takes place here. But, in this case, we know that there was a, a cut on the overland circuit. So all the there's a lot of the big submarine cable routes of the of the global internet connect Europe to Asia going through the Mediterranean and then somehow making their way through Egypt the Red Sea, out to the Indian Ocean and on to India Singapore and so on. And if people want a a picture of this, they can look at your blog post, just Google Doug Madurray and Yeah. I I well, I had an old one from a a telecom Egypt presentation. I used to go to a lot of submarine cable conferences.

Talk about, partly what we could see from internet measurement on submarine cable incidents. And so that was a one of the first tasks that was handed to me when I joined Renesys you know, we were twelve plus years ago was, like, we'll see what you can do. People people always ask us about submarine cables, see if you can find them. So I started to try to figure this out. And then, I mean, I it's still it's still a, a very, imprecise, process of, inference But, but you can you can get pretty good at it. But, anyway, so then, these cables, these large, large, seamui four, seemingly five, seemingly we stands for Southeast Asia, Middle East, Western Europe because the, acronym, the flacronym, Those are consortium cables, where, a bunch of different telecoms band together.

All those were led are led by Singtel is, is kind of the head honcho, but Any contour writers in there or are these all tough? Let's see. I guess you'd have to if you went and checked the submarine cable that severing cable map dot com, they usually list the owners. Okay.

And so, probably starting with CMMe five, but there's content providers. It was like a it was a a day. Yeah. The goats in in in the meadows, you know.

Everybody showed up, and they all started, driving the submarine cable industry, which considering its importance to the global internet and global economy, these are projects that are, you know, sometimes a billion dollars to put in a cable, you know, from one far away place to another, a lot of work goes into this. And, and the the the payoff, the ROI is actually considering the risk that's involved, it's not super attractive. So it ends up being a tough it's a tough go people don't need it, make it happen. Yeah.

Right. So then, you know, you need a consortium of people who are gonna be the captive audience the captive audiences, the ones put up the money to build this thing because they absolutely need it. But it's always like this and and summary cable conferences, there's always a discussion about building business case, models are trying to to find, the expense. And then all of a sudden content providers show up and just with, like, endless resources to build anything, and and and so then, you know, there's a lot of hand wringing around, industry around, you know, this a good thing or a bad thing?

Are they just, you know, crowding everybody out or are they, I mean, when people are talking about balloons and blimps for internet, is the one hand in which it's it's great to provide people access. There's the other in which the cynics would say, well, that's because there are some people that think it's unfair if somebody in the world can't click on an ad. So, you know, somewhere in between is truth. Or let me let me put another, you know, idea is that, let's say, because it's cool, they're gonna fly balloons into country X.

Well, there was a it was a startup that was trying to make a profit at creating a local business to provide service. And now they're up against these balloons that have no, they don't have to turn a profit. They don't have to do anything.

And they'll just operate until they find it's not Google anymore and just leave. In the meantime, this this this company goes out of business and these guys go bankrupt. So there's there can be some of these sometimes these unintended, consequences. Let's try to let's try to get them on a future podcast.

It's always interesting. We need still need to get Jared with his local ISP, and we got Elliot Noss. Oh, sure. I need to follow-up with him about, you know, his I talk with him about why would you start an ISP nowadays But, there's some cool stuff.

That'll be that'll be a great conversation. I would love to make one and Anyway, sorry. So back to Egypt.

Yeah. So What countries, you know, performed? Right. So let's see.

We knew there was a a major submarine cable outage.

Affecting, basically countries in East Africa, the Middle East, South Asia, is where, you know, it was showing up in in our traffic stats.

In the end, it was a an overland circuit over, Egypt. So the way all these cables go is, ships will go through the c o s canal, but the the submarine cables don't go through the canal. They have to come up on land. And they go either, there's a couple of routes, that come up, in Abu Tallada, Alexandria, or the, port cities, and then the Mediterranean, they cross the desert, and then then crop back in in the red sea, or, there's also an option that you can, run it in the the cement casing on the side of the the canal itself you know, in the water, but you're you're up out of the, the ground or out of the water. But, there's a route.

Right. If there's a break, you can get into it. But, so everything comes out of the water, and then it goes back into the water. And so because it's out of the water, that means if something breaks, there's a fix, can, you know, you can send a technician over there.

And it's a matter of hours versus, you know, it could be days or weeks if it was, in a deep sea location for the submarine cable. Yeah. But, and so in this case, again, it was it was something that was on land. They could get a fixed connection, a technician there to fix it.

Then that was just a matter of, I think, something on the order of four hours. But we could see, yeah, a lot of different countries, impacted. Others could too. So I mentioned in the blog, just my other, outfits that or other internet watchers, like me.

We try to call this what we are unique, view into this. And so I did pull up one. So Uberu, which is, cutter telecoms new branding as of that's probably, like, a decade old now, but that one I thought was the nice cleanest view. It's in the blog post of a a visualization of the what the lots of the kit, of this circuit knocked out a couple of their transit providers.

There was a shift. You know, they they basically stayed online, but but it it got they got rejumbled, over four hours, to to rejigger their transit to stay online. Mhmm. The other insight that, you know, it's kind of flipping around.

I was like, well, you know, we do a lot of cloud measurement. We do, at Kenix. We do a lot of synthetic performance monitoring.

Between the various cloud regions of all the the, you know, the major cloud providers. And so, I know just doing this in the, from in the past that, if you see a route like this that's going, you know, through through Egypt, through the Mediterranean, does the cloud providers have to use the same stuff? Mhmm. You know, normal people that you and I, have to use, we all have to use the same He do sharm of, IBM says without the network, there is no cloud.

Or Yeah. He alternatively says network is the water of the clouds. So Okay. I mean, yeah, there's there's been beautiful, beautiful metaphors, but I was thinking I was like, well, I I know because we looked at this.

I think a year ago, there was something like this where we also noticed I was like, I only go check and look at all of our routes going across Yeah.

Between any cloud region in Europe and cloud region in South Asia and sure enough.

There was we could see the impact fairly well. And, what was interesting was that there's different different impact by cloud provider. And, and so based on our stats, by sending our data, you could see the initial blip for AWS connectivity across this route, but it resolved itself fairly quickly. Azure seemed like it had a a period of a couple hours of higher latencies, but no packet loss.

But but Google Cloud Like, they didn't weather this one that well. No matter which direction we went, we saw, higher packet loss, higher latency, So that's, that's interesting. So that has to do with how how the how the cloud you know, contracts, it's it's routes, and how well it can fail over to backup, connectivity.

And, you know, for these for these links.

You're responding a lot of stuff that was falling over. Well, it's interesting because, you know, the first question I was gonna ask about this is it can be difficult for people that are not in in the routes and the packets and the bits and the bytes to understand that In some ways, connectivity can be much more network topological than, you know, geographically topological. It's like how the networks connect can affect performance and routing outages and things much more. So just because two things are in a country doesn't mean they're gonna have good performance or, you know, just because something is three countries away, doesn't mean it'll have worse.

So, you know, when you look at what happened, with the with the circuit outage where there, you know, if you look for the non cloud, where there's some you know, were any countries entirely cut off, or was it really just provider by provider forgiven? So if there's fewer providers and one big one is affected, that affects more of it, but doesn't knock out all the all the traffic to a country. It's like, were there any countries completely cut off, or was it a somalia, but that's, they're kind of a a situation is really only one summary cable, serving them the Okay. The easy cable, and, they have, yeah, just limited kind of satellite, you know, that that we may not geo.

I mean, they don't when these, what what I've seen over, you know, over the last ten years, ten plus years, when a submarine cable arrives, the market for bulk satellite just go just dries up. I mean, we saw this in Tonga just in January, I mean, the the volcano, you know, had a dish been up and ready to go. It might've got knocked over. Anyway.

But it but it wasn't it wasn't ready to go. They they had long since, gotten rid of their, backup satellite, which was how they had stayed online prior to that submarine cable coming online. Yeah. And so we'd see this all over, Africa where countries, as soon as the submarine cable, came aboard, or came ashore, everybody switched over.

The satellites gone. You're not going to give it to the home on on, you know, twenty four megabit uplink by satellite. So, I mean, yeah, there's a capacity thing, but I but also they don't even they don't even seem to keep it around. It's still it's even too costly a dormant backup, which just seems to be the case.

It is very expensive, but Yeah. Leo stuff is making it less expensive.

So then if I could make a further leap, If you go to the Snow crash view, but, you know, the cyber the dystopian cyberpunk view, in essence, the large tech companies, the cloud providers will be the countries of the future. So countries in the future Maybe we're already there. Well, we just don't know it with states that we don't cover.

So maybe, you know, if there's two kinds of of country wide impact. One is the internet countries and one is the physical countries. And, you know, I mean, these are large communities that need collaborate. And so, they also and I know they all do work very hard at finding, you know, redundant multipath connectivity and looking performance and, you know, all these things. That's true. I mean, I I wanna disparage anybody. These guys are the best, that they have a a lot of smart people, a lot of resources So it was actually kinda surprising to see, the severe, issues that our our data was showing, for Google.

So I guess the last last, question topically. So, you know, I saw I saw your tweet about you know, it's exam time in Syria again. Let's shut the internet down. And I was just thinking, is this is this really the new norm, are we just gonna be seeing this?

Is this full employment for you and others? Is this, do you do this? Is there anything gonna change? This is one that's hard to get your mind around.

I do I even just I just explained this. I think at this conference, in Bond, Germany. Again, people are like, What is this? Or I just can't kinda understand this.

And, again, for those who are unfamiliar with this, the situation is So take a rack, which was the first one that I started reporting on in, I think, twenty fifteen. So at that time, you imagine that ISIS has taken muscle in this threatening bag that, there's, like, a heat wave, riots for corruption, all these things are happening. And the internet goes down. And I had a a well placed contact in the Iraqi Iraqi government.

And so I'm reaching out to him. I said, what what's happening here? Is that you're not gonna believe this. It's, it's student exams.

I was like, I you're gonna have to explain that to me. Like, I I assumed it was something there's all these huge things happening in this country, and that's what they turned the internet off for. But here's the explanation as, you know, in a country like, Iraq.

You know, these are, this national test at sixth grade. If you don't score high enough, that's the end of your public education.

And, and to be a There's a surprising number of countries where there's something like that that goes on. Yeah. I mean, it it's it's, you know, I don't know if you for anybody who read the the book freakonomics for a year years back when, discussion of when you make the, the consequences so severe you justify you end up justifying cheating. You're justifying, extreme measures to avoid these extreme consequences.

And so in this case, If you are if you are a parent, in one of these countries, your kids have got a lot of, cards stacked against you. A guy, I guess them. And, and so they wanna get any any kind of, help they can. And so then it ends up happening is over time the tests get compromised cheating becomes rampant and, and so then this is like a a last ditch effort to try to regain control over, a system that's gotten out of hand, but it's not, you know, I guess it wasn't didn't happen in Iraq this year, but, yeah, Syria, Sudan, I I someone reached out to me to ask me to look into Algeria, which I guess, there's doing something similar.

Sometimes it's, you know, the whole country's down or somebody's just mobile service. Right. The way the way it works in Syria is that they, they take off, take down the the back backbone of the, the, of the country while they physically distribute the tests. And then when they bring, they bring the backbone back up and the mobile service comes down while the kids are taking a test.

Anyway, this is what This is what Better. I mean, you would think a faraday cage would be better, but no.

Yeah.

But it's that's what And there's no I mean, you follow some of the policy stuff. There's no, you know, there's no UN group talking about internet as a human right and trying to form a treaty, you know, basically saying that internet won't be used for So there are, yeah, there are digital rights organizations to fervently put fervently push back on this. Access now is the one that probably is argue that maybe the most prominent, their organization that we kind of, support with some technical consultation.

I I try to, let those folks know about things. For example, when when this, submarine cable, occurred a couple weeks ago, that's now on my list is to let know. Let some of the folks in the digital rights space know that something happened, with that's infrastructure based and not, government directed shutdown just so they because they're gonna start, receiving, complaints or receiving, people claiming that there's, a shutdown or something.

And so I that's on that's a high up on my list is to let those folks know. Like, already something just happened, and it's not this is something broke. Like, it wasn't a government directed thing.

But, I know that they try to push back, but in the end, you know, we still live in a westphalian world of, countries are absolute sovereigns. And they if they wanna make a bad decision, they are they can do so. We're fortunate that internet governance itself has survived internationalization, I'll just say. I'm not saying it's necessarily good for you.

So far so far. So far. I'm not saying it's necessarily good for any one company, a country you know, to to, you know, be the only, but, yeah, when things turn to politicking and interests and especially with the value of IP space, you know, and demands being what it is. You know, it's, good that things are working as well as they are.

So Any other, any other, hot topics things going on?

Connecting the infrastructure and news or just infrastructure wise?

I think that's all I've got for today.

Well, I actually do hope it'll be a calm next few months. And, history, especially with what we're going on, you know, might not If history projects forward, you know, you might not think that, but, I certainly hope that. And, we'll do an update to this, as there is more news.

Sounds good. Thanks, Doug, for joining, and so late from, from Germany.

No problem. Thanks, to the audience for listening to network a f in this episode. You can find us on Apple Podcasts at our web page and your favorite podcast listings, around you. And, Doug, people are curious to contact you? How can they do that?

Yeah. Usually, you can follow me on Twitter and send me a message. I if you get me going, I love talking about this stuff. So Avi can vouch for that.

Absolutely. We've talked for hours and hours. And, I'm Avi at kintech dot com. I'm Avi friedman on Twitter and LinkedIn.

Thanks again.