Beyond Their Intended Scope: BGP Goof-UPX
Summary
In this second installment of Beyond Their Intended Scope, we analyze a recent BGP leak out of Brazil that briefly affected networks around the world. Because this routing mishap was a path leak (i.e., did not involve any mis-originations and therefore immune from RPKI ROV protection), it demonstrates why we need a thing called ASPA … ASAP.
In October, we started a new blog series entitled Beyond Their Intended Scope, intended to shed light on BGP mishaps which may have escaped the attention of the community but are still worthy of analysis.
In this follow-up installment, let’s take a look at a recent routing mishap that briefly disrupted and misdirected internet traffic from around the world through Brazil, a country that has emerged as arguably the top source of BGP leaks in recent years.
It may come as a surprise to many that route leaks continue to occur with some regularity. The difference today is that routing hygiene has improved to such a point that these leaks are often contained to the country or region where they originated, limiting the disruption. Despite this, they are still important to study as they can help us better understand what is and isn’t working in routing security.
What are BGP leaks?
“A route leak is the propagation of routing announcement(s) beyond their intended scope.”
That was the overarching definition of a BGP route leak introduced by RFC7908 in 2016. Border Gateway Protocol (BGP) enables the internet to function by providing a mechanism by which autonomous systems (e.g., telecoms, companies, universities, etc.) exchange information on how to forward packets based on their destination IP addresses.
In this context, the term “route,” when a noun, is shorthand for the prefix (range of IP addresses), AS_PATH, and other associated information relating to packet delivery. When routes are circulated farther than where they are supposed to go, traffic can be misdirected, or even disrupted, as happens numerous times per year.
RFC7908 went on to define a taxonomy for BGP leaks by enumerating six common scenarios. In my writing on route leaks, I like to group them into two broad categories: mis-originations and path errors. As I described in my blog post last year, A Brief History of the Internet’s Biggest BGP Incidents, this distinction is useful because the two types of error require different mitigation strategies.
- A mis-origination occurs when an AS originates (announces with its ASN as the origin) a new advertisement of a route to an IP address block over which it does not possess legitimate control, consequently soliciting traffic destined for those IP addresses.
- An AS path error occurs when an AS inserts itself as an illegitimate intermediary into the forwarding path of traffic bound for a different destination.
With those definitions out of the way, let’s get into the details of this particular incident.
What happened?
Beginning at 18:16 UTC on February 11, 2025, AS52863 (UPX Technologies, Brazil) began leaking routes from one provider (EdgeUno, AS7195) to a sibling network AS400282 (UPX Technologies, US) and on to its US providers.
First reported by our friends over at Qrator, the incident involved the leak of over 20,000 routes lasting around 15 minutes, misdirecting and disrupting traffic to providers around the world.
| Mis-origination or path error? | Path error |
| How many leaked routes? | 21,119 (4,295 seen by 50+ Routeviews peers) |
| New more-specifics? | 0 (normally seen by 10 or fewer Routeviews peers) |
In this post, we’ll take a closer look at what happened, the impact on traffic, as well as what can be done to prevent these types of incidents in the future.
Impacts
BGP
As we did last time, let’s start at the individual BGP message level and work our way out from there. The two BGP messages below, collected from Routeviews, show the AS paths before and during the leak from the perspective of one BGP source, Norwegian provider Nexthop (AS49788).
Reading the highlighted lines from the left, we can see that, in both cases, AS49788 receives the route for 31.56.24.0/24 (Australian network OneQode) from its transit provider Arelion (AS1299). However, during the leak, Arelion switches to receiving the route from GTT (AS3257) instead of directly from the origin OneQode (AS140627).
TIME: 02/11/25 18:00:01 FROM: 91.218.184.60 AS49788 ORIGINATED: 02/08/25 23:04:01 ORIGIN: IGP ASPATH: 49788 1299 140627 140627 140627 NEXT_HOP: 91.218.184.60 COMMUNITY: 1299:37000 ANNOUNCE 31.56.24.0/24
BGP announcement containing the AS28910 leak:
TIME: 02/11/25 18:16:42.905577 FROM: 91.218.184.60 AS49788 ORIGIN: IGP ASPATH: 49788 1299 3257 400282 52863 7195 140627 NEXT_HOP: 91.218.184.60 COMMUNITY: 1299:20000 ANNOUNCE 31.56.24.0/24
It is curious that the leaked route with a longer AS path is preferable to the more direct route — path length is only one factor of BGP route selection. It probably doesn’t help that the normal route is prepended twice to its main transit providers, NTT (AS2914) and Arelion (AS1299), incentivizing other networks to use an alternative route if one appears.
The last clue is in the BGP communities that are visible in these messages. In this case, these communities are informational, just telling us about the route, as opposed to directive, (e.g., “Do not announce to All Peers in Asia & Pacific”). The community changes from 1299:37000, reserved for AS1299’s Asia+Pacific Customers, to 1299:20000, reserved for AS1299’s EU Peers.
So, during the leak, AS1299 opts for the longer AS path to a peer versus the shorter (albeit prepended) AS path directly to its customer, which is doubly counterintuitive. Nothing sinister; it’s just sometimes hard to know all of the factors that go into route selection.
Now, let’s zoom out and see what this routing change looks like in the aggregate. Using Kentik’s BGP visualization, shown below, we can see how the internet reached AS140627 for 31.56.24.0/24 around the time of the leak.
While the lower portion of the visualization shows a pruned ball-n-stick AS-level diagram, the upper graph depicts the ASes observed upstream of AS140627 for this route by count of BGP vantage points. The evidence of the leak is marked in orange boxes.
Let’s look at another example visualized below.
57.144.128.0/23 is announced by Facebook (AS32934) out of Brazil and is normally only seen by about 22.3% of BGP sources in Routeviews. The limited propagation is normal for what are sometimes called “regional routes,” or routes that are only intended to direct traffic for a part of the world.
The problem for regional routes is that when someone leaks these routes, they spread widely due to the lack of contention as a result of the limited propagation. This manifests as the hump in the stacked plot below as AS7195 temporarily appears that the world’s most popular upstream of AS32934 for this and the other Facebook routes with intentionally limited propagation.
Additionally, since these regional routes are often more-specifics of less-specific routes (used to handle traffic outside of the region), they tend to attract a lot of traffic, leading to traffic disruption. Regional routes are risky for that reason.
Traffic
One of the amazing things that Kentik has the ability to do is to take a BGP incident and analyze its impact on Internet traffic. Using Kentik’s unique aggregated NetFlow data, we can peer into the impact on internet traffic, specifically how much traffic was misdirected vs dropped entirely.
During ingest, Kentik annotates each NetFlow record with the AS path of the destination IP from the perspective of the router producing the NetFlow. This allows us to then make a query that retrieves NetFlow records which were annotated with AS paths that match the leak.
Let’s first take a look at all of the traffic that was sent along AS paths with the subsequence “3257 400282 52863 7195” by the top destination AS network. Running that query across our aggregate NetFlow yields the following graph with Claro (AS28573), Sim.Digital (AS268207) and Cloudflare (AS13335) as the top three impacted networks in bits/sec.
When we look at all of the traffic destined for IP space contained in the top leaked routes, we can gain a sense for how much traffic was misdirected versus dropped altogether. (Note: in this case, I took the 1,868 leaked routes that were seen by at least 200 Routeviews peers.)
In the graphic below, a steady state of traffic flows from our customer’s networks to these routes. During the leak, the overall amount of traffic drops due to its disruptive effect. Marked along the bottom, we can also see a smaller portion of traffic sent to routes with the leak in the AS paths. This is the portion of traffic that is misdirected but ultimately reaches its destination.
I have found that, often the case with routing leaks like these, the amount of traffic that is dropped or disrupted exceeds the amount that is misdirected. Traffic gets dropped during leaks because an unexpected volume of traffic is directed down a link that isn’t adequately provisioned to handle it.
We don’t know where the bottleneck was exactly along the leak path (“3257 400282 52863 7195”), but likely, something got overwhelmed, and couldn’t deliver packets where they needed to go in a timely manner.
Call to action
The BGP leak analyzed in this post can be referred to as a “path leak” or “adjacency leak,” meaning that the mistake occurred in the middle of the AS path, not at the beginning (rightmost ASN). And since no routes were mistakenly originated in a problematic way, origin-based solutions like RPKI ROV would not be in a position to help.
Autonomous System Provider Authorization (ASPA) was created to address the path leak scenario. Like ROV, ASPA is a routing security technology that will make use of existing RPKI infrastructure.
ASPA works by having ASes assert their transit relationships as records in the RPKI. This information enables other ASes to reject routes from path leaks by evaluating the AS paths as RPKI-invalid — in violation of the transit assertions in ASPA records.
But, ASPA is still in its infancy. We need more ASes to create ASPA records, and to do that, we need an easier workflow like we have for the creation of ROAs — something the RIRs are working on presently. Additionally, we need more greater support from equipment vendors to add the logic of ASPA to their process of evaluating routes for rejection.
These things take time and persistence. The significant progress made in the global adoption of RPKI ROV teaches us that careful design, when coupled with education and advocacy, pays off — but it can take years. If we want to address this vulnerability in BGP routing, there is no other way but to do the work.
For more information about ASPA about how it works, see Kotikalapudi Sriram’s talk at NANOG 83.
