Beyond Their Intended Scope: Uzing into Russia
Summary
The first installment of our new blog series, Beyond Their Intended Scope, covers BGP mishaps that may have escaped the community’s attention but are worthy of analysis. In this post, we review a recent BGP leak that redirected internet traffic through Russia and Central Asia as a result of a path error leak by Uztelecom, the incumbent service provider of Uzbekistan.
Welcome to a new blog series entitled Beyond Their Intended Scope which intends to shed some light on BGP mishaps that may have escaped the attention of the community but are worthy of analysis.
In the first installment, let’s take a look at a recent BGP leak that redirected internet traffic through Russia and Central Asia as a result of a path error leak by Uztelecom, the incumbent service provider of Uzbekistan.
It may come as a surprise to many that route leaks continue to occur with some regularity. The difference today is that routing hygiene has improved to such a point that these leaks are often contained to the country or region where they originated, limiting the disruption. Despite this, they are still important to study as they can help us better understand what is and isn’t working in routing security.
What are BGP leaks?
“A route leak is the propagation of routing announcement(s) beyond their intended scope.”
That was the overarching definition of a BGP route leak introduced by RFC7908 in 2016. Border Gateway Protocol (BGP) enables the internet to function by providing a mechanism by which autonomous systems (e.g., telecoms, companies, universities, etc.) exchange information on how to forward packets based on their destination IP addresses.
In this context, the term “route,” when a noun, is shorthand for the prefix (range of IP addresses), AS_PATH and other associated information relating to packet delivery. When routes are circulated farther than where they are supposed to go, traffic can be misdirected, or even disrupted, as happens numerous times per year.
RFC7908 went on to define a taxonomy for BGP leaks by enumerating six common scenarios, half of which appear in the two leaks covered in this post. In my writing on route leaks, I like to group them into two broad categories: mis-originations and path errors. As I described in my blog post last year, A Brief History of the Internet’s Biggest BGP Incidents, this distinction is useful because the two types of error require different mitigation strategies.
- A mis-origination occurs when an AS originates (announces with its ASN as the origin) a new advertisement of a route to an IP address block over which it does not possess legitimate control, consequently soliciting traffic destined to those IP addresses.
- An AS path error occurs when an AS inserts itself as an illegitimate intermediary into the forwarding path of traffic bound for a different destination.
With those definitions out of the way, let’s get into the details of this particular incident.
What happened?
Beginning at 06:25 UTC on September 26, 2024, Uztelecom (AS28910), incumbent of the former Soviet republic of Uzbekistan, began leaking routes from its peers through one of its transit providers, Rostelecom (AS12389), Russia’s state telecom. First reported by our friends over at Qrator, the incident involved the leak of over 3,000 routes lasting about 40 minutes misdirecting traffic from a dozen countries.
| Mis-origination or path error? | Path error |
| How many leaked routes? | 3144 (371 seen by 50+ Routeviews peers) |
| New more-specifics? | None |
In this post, we’ll take a closer look at what happened, the impact on traffic, as well as what can be done to prevent these types of incidents in the future.
Impacts
BGP
Let’s start at the individual BGP message level and work out from there.
In the two BGP messages below collected from Routeviews, we can see AS14041 (University Corporation for Atmospheric Research) in the United States initially accepting an Amazon route from its transit provider Arelion (AS1299). During the leak, AS14041 began selecting the leaked route with a longer AS path from a peer Hurricane Electric (AS6939). The leaked route is likely preferable because of a localpref setting which would prefer sending traffic for free through a peer regardless of the AS path length, over paying to send traffic through a transit provider.
TIME: 09/26/24 06:00:12 FROM: 192.43.217.142 AS14041 ORIGIN: IGP ASPATH: 14041 1299 174 16509 NEXT_HOP: 192.43.217.142 COMMUNITY: 1299:25000 14041:102 ANNOUNCE 185.2.49.0/24
BGP announcement containing the AS28910 leak:
TIME: 09/26/24 06:28:49.293030 FROM: 192.43.217.142 AS14041 ORIGIN: IGP ASPATH: 14041 6939 12389 28910 8359 16509 NEXT_HOP: 192.43.217.142 COMMUNITY: 14041:400 ANNOUNCE 185.2.49.0/24
Kentik’s BGP visualization, shown below, illustrates how routes from hundreds of BGP vantage points changed for this Amazon prefix during the leak. While the lower portion of the visualization shows a pruned ball-n-stick AS-level diagram, the upper graph depicts the ASes observed upstream of Amazon’s ASN (AS16509) for this route by count of BGP vantage points. The evidence of the leak is marked in red boxes.
During the leak, VEON (AS3216) suddenly emerges as a popular upstream, as it is a transit provider for leaker Uztelecom (AS28910). Traffic traveling along the leaked route could either be misdirected through Russia (AS12389) and Uzbekistan (AS28910) or simply go undelivered due to congestion or excessive latency.
Another prefix sucked into the leak (162.158.84.0/24) belonged to Cloudflare. As depicted in the upper graph of the visualization below, this prefix is normally present in the tables of just over half of our BGP sources — likely a regional route with intentionally limited propagation to steer traffic in only this part of the world.
As was the case in the previous example, AS3216 appeared as a new popular upstream of Cloudflare (AS13335) for 162.158.84.0/24 as AS28910 began leaking the route through AS12389. Leaks are especially impactful to routes with limited propagation like this one because, for much of the internet, there is no alternative version for the leaked route to contend against, making it an automatic winner in the BGP selection algorithm.
Hosting provider xTom also had routes impacted by this leak, including 103.135.234.0/24 shown below. Like the previous example, the leak appears in the upper graph as a new popular upstream to xTom, Russian fixed-line operator Transtelecom (AS20485) — another network providing transit (albeit indirectly) to AS28910. Despite its AS path length, the leaked version of the route becomes quite popular during the 40-minute leak.
Traffic
One of the unique things that Kentik has the ability to do is to take a BGP incident and analyze its impact on internet traffic. Using Kentik’s unique aggregated NetFlow data, we can get a sense for how much traffic was misdirected vs dropped entirely.
During ingest, Kentik annotates each NetFlow record with the AS path of the destination IP from the perspective of the router producing the NetFlow. This allows us to then make a query that retrieves NetFlow records which were annotated with AS paths that match the leak.
Below is the result of a query which retrieved aggregate NetFlow directed along AS paths containing “12389 28910” (Rostelecom to Uztelecom) — excluding traffic destined for Uzbekistan, since that’s the majority case and to be expected.
What results is a fascinating shift in the makeup of the traffic along 12389_28910 during the leak. The impact is clearly evident as the traffic goes from being solely destined for Tajikistan and Turkmenistan, to also destined for countries outside of Central Asia: Hong Kong, the Netherlands, Afghanistan, Russia, Japan, and the United States.
Call to action
Years ago large routing leaks like these might have been the cause of widespread internet disruption. Not so much anymore.
Humans are still (for the time being) configuring routers and, being human, are prone to the occasional mistake. What has changed is that the global routing system has become better at containing the evitable goof-ups. Route hygiene has improved due to efforts like MANRS and the hard work of network engineers around the world.
In September 2024, the White House Office of the National Cyber Director released the Roadmap to Enhancing Internet Routing Security, which aims to address BGP vulnerabilities. The report cited my recent collaboration with Job Snijders of Fastly on RPKI ROV adoption, a technology that aims to reduce the disruption caused by things like BGP route leaks.
That progress is largely the macro-level, but there are many individual networks that have not deployed RPKI and to them I would say the following:
- Creating ROAs will help to protect your inbound traffic by asserting to the rest of the internet which origin is the legitimate one.
- Conversely, rejecting RPKI-invalids helps to protect your outbound traffic by rejecting leaked routes that might misdirect that traffic or lead to a disruption.
By continuing to reduce the impact of BGP leaks, we can focus on the harder problems left to be solved in routing security. Problems such as the “determined adversary” scenario witnessed in last year’s attacks on cryptocurrency services. In that realm, there is still much work to be done.
