A Year in Analysis: 2024

Summary

In this post, Doug Madory reviews the highlights of his wide-ranging internet analysis from the past year, which included covering the state of BGP (leaks and the state of RPKI adoption), submarine cables (both cuts), major outages, and how geopolitics has shaped the internet in 2024.

It’s the end of another eventful year on the internet and time for another end-of-year recap of the analysis we’ve published since our last annual recap. This year, we’ve organized the pieces into a few categories, including BGP analysis, major internet outages, submarine cables, and geopolitics. We hope you find this annual summary insightful.

Border Gateway Protocol (BGP) analysis

This year, I continued my ongoing collaboration with internet routing expert Job Snijders of Fastly, looking at the state of RPKI ROV (Route Origin Validation) adoption. In May, we covered a major milestone as the majority of routes in the IPv4 routing table now have Route Origin Authorizations (ROAs) — IPv6 achieved this milestone last year.

Now that the majority of both IPv4 and IPv6 routes in the global routing table have ROAs, we can update our traffic statistics, which show that we’re approaching three-quarters of all traffic (measured in bits/sec, pictured below) heading to routes with ROAs and thus eligible for the protection that RPKI ROV provides. I’m not sure for how much longer we can maintain this steady rate of adoption, but it has been remarkable thus far.

Job and I also got into the weeds by delving deeper into the cryptographic operations of RPKI to explain why ROAs have “effective expirations” that are perpetually about to expire. As colorfully illustrated below, the effective expiration of a ROA is heavily dependent on which RIR publishes it due to differences in software implementations.

Additionally, this year, the US Federal Communications Commission made its first moves towards improving BGP routing security by publishing proposed rules that would require the nine largest telecoms in the US to deploy RPKI ROV. In a post from July, I reviewed where each of these companies stands with respect to this proposed new requirement and the debate around mandating the use of RPKI ROV.

Lastly, a new challenge that I believe we face in the routing security community is that we may become victims of our success. Specifically, that routing hygiene has improved to the point that headline-grabbing BGP mishaps are becoming rarer, potentially taking wind out of the sails of various advocacy efforts like MANRS. The improvements are, of course, good news and have many contributing causes (increased adoption of RPKI ROV, greater levels of IRR-based route filtering, greater awareness of routing issues, peerlock), but there are numerous unsolved issues around BGP security that will need continued attention.

That fact is that route leaks continue to occur, even if the potential disruption they could cause has often been mitigated. I began a new blog series in October entitled Beyond Their Intended Scope to shed some light on the BGP mishaps that may have escaped the attention of the community but are worthy of analysis. The first edition covered a leak that redirected internet traffic through Russia and Central Asia as a result of a path error leak by Uztelecom, the incumbent service provider of Uzbekistan.

Major internet outages

The first outage analysis piece I wrote this year covered the outage caused by RPKI ROV due to the compromise of Orange España’s RPKI NCC account. Spain’s second-largest mobile operator suffered a national outage as a result of a hacker’s unprecedented use of RPKI as a tool for denial of service. Using a password found in a public leak of stolen credentials, a hacker was able to log into Orange España’s RIPE NCC portal using the password “ripeadmin.” Oops!

I concluded that piece by saying that while RPKI was employed as a central instrument of this attack, it should not be construed as the cause of the outage any more than we would blame a router if an adversary were to get ahold of the login credentials and started disabling interfaces. In my opinion, the silver lining of the outage was that it showed that RPKI-invalid routes really do get rejected at a significant rate. Otherwise, the outage could not have occurred.

In a somber post, I also covered the effects of Hurricane Helene, which cost the lives of over 200 people in the Southeast United States. I combined our traffic data with internet measurement data from Georgia Tech’s IODA tool to survey the disruption of internet connectivity in the region.

Submarine cable failures

It has been another eventful year in the world of submarine cables. The two highlights of our coverage this year were our collaborations with WIRED magazine and The New York Times.

In the spring, we teamed up with WIRED to investigate the cause of the submarine cable cuts in the Red Sea in February. As you may recall, in response to the ongoing war in Gaza, the Houthi-controlled Yemeni government began firing missiles and armed drones at ships transiting the nearby Bab al-Mandab strait that they believed had an affiliation with Israel, Britain, or the United States.

As I wrote In my blog post, on February 24, three submarine cables were cut in the Red Sea (Seacom/TGN-EA, EIG, and AAE-1), disrupting internet traffic for service providers from East Africa to Southeast Asia. Initial speculation of the cause of the cuts focused on the purported threats against submarine cables posted in a Telegram channel months earlier.

Before long, a more realistic theory emerged from the submarine cable industry. Days before the three subsea cable failures, a Belize-flagged, United Kingdom-owned cargo ship was struck by missiles fired from Yemen. The crew dropped anchor and abandoned the crippled ship, the MV Rubymar. Afterward, the Rubymar began to drift, dragging its anchor — one of the top causes of submarine cable cuts, according to the International Cable Protection Committee. On March 2, the derelict vessel finally sank, taking with it more than 41,000 tons of fertilizer.

WIRED combined our data with AIS location tracking data and satellite imagery to estimate the location of the Rubymar and place it near the location of the cable cuts.

The following month, after the submarine cable cuts in the Red Sea, another multi-cable incident took place on the opposite side of the continent of Africa. On March 14, an undersea landslide in the Trou Sans Fond (French for bottomless pit) Canyon off the coast of Abidjan, Côte d’Ivoire.

I worked with the investigative team at the New York Times as they researched the incident from its impact to the crews tasked with repairing severed undersea cables at great depths. The resulting piece was a beautifully written and presented story of the outages and repairs that took place this year. It’s really worth your time if you haven’t read it.

Geopolitics

In August, I wrote a follow-up piece to my 2021 post written in the aftermath of the US withdrawal from Afghanistan. This led me to get connected with the organizers of the newly formed AFNOG (Afghanistan Network Operators Group), who continue to work to develop internet service in the war-battered nation despite an array of unique challenges, from austere terrain to international sanctions.

I was honored to be included in the opening ceremony of this year’s AFNOG2 and served as a moderator for a fascinating discussion on navigating the country’s transit and peering environment.

Over-the-top anatomies

Kentik’s OTT Service Tracking (part of Kentik Service Provider Analytics) combines DNS queries with NetFlow to allow a user to understand exactly how OTT services are being delivered — an invaluable capability when trying to determine what is responsible for the latest traffic surge. Whether it is a Call of Duty update or the first-ever exclusively live-streamed NFL playoff game, these OTT traffic events can put a lot of load on a network and understanding them is necessary to keep a network operating at an optimal level.

To demonstrate some of the unique capabilities of the OTT Service Tracker, I occasionally publish blog posts as part of a series called Anatomy of an OTT Traffic Surge. This year I covered:

• The challenges Netflix faced during its live event streaming the highly anticipated, if somewhat absurd, boxing match between the 58-year-old former heavyweight champion Mike Tyson and social media star Jake Paul.
• The surge of traffic triggered by the release of Fortnite’s Chapter 2 Remix featuring rapper Eminem and special NBC Olympics correspondent Snoop Dogg.
• A dissection of the delivery of one of Microsoft’s Patch Tuesday software updates.
• The first-ever exclusively live-streamed NFL playoff game delivered while refraining from making any references to pop superstar Taylor Swift or her sizzling romance with nine-time Pro Bowler Travis Kelce.

As depicted below, this capability gives us the In aggregate the top individual OTT services we see providers include Netflix and Youtube at the top spots followed by social media stalwarts Instagram, Facebook, and soon-to-be-banned(?) Tiktok.

Cloud Latency Map

In October, we released the Kentik Cloud Latency Map, a free public tool that employs Kentik’s synthetics capabilities to allow users to explore the latencies measured between over 100 different cloud regions located around the world.

As I said in my write-up announcing the release, the Map is the latest expression of Kentik’s dedication to extending network observability to the cloud.

Broadly speaking, the Map can assist someone trying to determine if there is a connectivity issue impacting particular cloud regions. Additionally, since the public clouds rely on the same physical infrastructure as the rest of the global internet, the Map can often pick up on the latency impacts of failures of core infrastructure, such as the loss of a major submarine cable.

Since its release, I have posted a handful of interesting observations found in the Map, including the temporary rerouting of traffic between AWS’s and Azure’s cloud regions in Sao Paulo, Brazil, as well as a significant change in connectivity between AWS in China and Google in both Europe and Asia.

As the Map illustrates, it’s a volatile internet out there — even for rarefied traffic between the hyperscalers. Take a look around and see what you can find. The data is updated every hour, and we have a list of features we’d like to add to it in the future.

Podcasts

If anything, 2024 might be the year of podcasts! I haven’t included this as a section in previous year-end write-ups, but this year was a real boon for audiophiles.

I was the subject of an episode of networking mainstays The Art of Network Engineering back in February and Imposter Syndrome Network in April. While at APRICOT 2024 in Bangkok, Thailand, I recorded a conversation with APNIC’s podcast PING about BGP, RPKI, and Oregon’s Routeviews project. I got deeper into BGP security with Chaos Lever and Russ While’s Hedge podcast.

In June, I was interviewed on Bitporto, the podcast of the Italian IXP NAMEX. Versions are available in English, and, in a first, I was also dubbed into Italian.

And, of course, I make the occasional appearance on Kentik’s Telemetry Now podcast hosted by my colleague Phil Gervasi. In four episodes this year, I discussed submarine cables and RPKI adoption. I also helped interview Andres Azpurua about Internet Censorship in Venezuela and out-going president (and my former colleague at Dyn) Andrew Sullivan on how the Internet Society helps maintain an open Internet.

Conclusion

To watch my webinar covering these and other topics, please check out the link below:

And if you enjoy reading these annual recaps, please check out our annual rollups from 2022, 2023, as well as our list of the top outages from 2021.