Using Kentik to Fight DDoS at the Source
Summary
In this blog post, we describe how one backbone service provider uses Kentik to identify and root out spoofed traffic used to launch DDoS attacks. It’s a “moral responsibility,” says their chief architect.
For decades, the scourge of distributed denial of service (DDoS) attacks has plagued the internet. One of the most common forms of DDoS attack is the reflection attack. This type of DDoS takes advantage of stateless protocols to trick a myriad of internet-connected devices into sending an overwhelming amount of traffic to an intended target, degrading its connectivity or completely knocking it offline.
The attacker sends thousands of requests with “spoofed” source IP addresses. Instead of the attacker’s real IP address, the source IP address is modified to be the address of the intended target. These requests utilize a stateless protocol like NTP or DNS. It’s typically a UDP protocol since the attacker wouldn’t be able to complete TCP’s three-way handshake with a spoofed source address, although some attacks have found ways around this restriction.
During the attack, thousands of servers unwittingly respond to these UDP requests with responses sent to the forged source address, which, of course, is the target of the attack. Incidents such as these take place numerous times every day and have given rise to the symbiotic industries of DDoS-for-hire services as well as the DDoS mitigation vendors that defend against their attacks.
One of the reasons that reflection attacks are so difficult to stop is that the victim has no idea where on the internet the attack is actually being initiated. While the victim network might see the IP addresses of the servers doing the reflection, it is unable to discern the true source from the attack traffic.
Addressing the problem
There are multiple places to break the chain of events leading to reflection-style DDoS attacks. First, networks should implement BCP38 where possible to do so. This will prevent the hosts on a network from being able to send spoofed traffic to the internet.
A BCP is an IETF Request for Comment document, which has been highlighted as recommending a Best Current Practice. BCP38 presently refers to RFC2827: Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing.
BCP38 specifies that a network should drop packets with source IP addresses that are likely forged. It determines that a packet has a forged source address if the address is not in the forwarding table of the interface it was received on. In other words, packets, which if forwarded on to the internet, would elicit a response that could never come back through that interface are therefore likely to be spoofed source addresses.
Despite the publication of BCP38 in the year 2000, many networks still allow spoofed traffic. In some cases, this is because BCP38 filters can block legitimate traffic, such as in the case of multihoming (although there is a BCP for that, as well).
Regardless of the reason, we simply can’t solely rely on BCP38 filtering to prevent spoofed traffic. We need service providers (SPs) to do their part as well. One way SPs can help is to police their network for spoofed traffic using a NetFlow analytics platform like Kentik.
Using Kentik to traceback DDoS sources
“Service providers have a moral responsibility to identify and remediate customer networks which are sending spoofed traffic,” says Aaron Weintraub, chief architect for Cogent Communications, one of the world’s largest network service providers.
Aaron utilizes a customized workflow in Kentik’s Data Explorer (KDE) to identify customer networks which are sending traffic in violation of BCP38. KDE is an extensive NetFlow analysis platform used by hundreds of companies to optimize and manage how traffic traverses their network.
His methodology boils down to two steps:
Step 1. Find spikes of packets from customer networks to a large set of unique destination IP addresses using commonly abused UDP ports. For this Aaron uses the following ports: 19, 53, 123, 161, 389, 427, 1900, 3283, 3702, 10001, 10074, 11211, 37810, 32414.
He excludes a list of customer networks that he has already investigated — either because the traffic turned out to be legitimate (e.g., a network hosting a popular DNS server), or because he is already actively engaging with the customer about the problematic traffic.
Aaron builds a query like the one below, using packets/sec and unique destination IP address count as the metrics. Then, he plots the results on a line chart for visual inspection. To expedite this process, he has these queries saved in a library, so they are trivial to rerun.
Step 2. For any suspicious spikes in packets to those selected UDP ports, a quick subsequent investigation reveals the nature of the traffic. He next checks the source IPs of these packets coming from that customer.
Aaron runs a query to isolate traffic coming from the router interface(s) serving the customer. He’s looking for the telltale signs of spoofed traffic responsible for launching DDoS attacks.
That traffic will have “source IPs” from an improbably diverse set of ASNs — networks which would have never routed traffic through the customer network to Cogent. This diverse set of ASNs are actually the targets of on-going DDoS attacks that are being launched by this spoofed traffic.
Below is an example of the secondary analysis described above. It shows spikes of packets across a customer interface to thousands of unique destination IPs (the unwitting reflection nodes) and seemingly from a variety of source networks, such as Kazakh mobile provider Beeline and US mobile provider Verizon, the targets of the DDoS attacks.
Now the fun begins
Finding a customer network that is the source of spoofed traffic triggering DDoS attacks might actually be the easy part (provided one has Kentik) — getting them to stop? Well, that is a different animal entirely.
Aaron could simply refer the traffic to Cogent’s abuse team for them to take action according to their acceptable use policy (AUP) that they, as well as every customer agreed to upon initiating service. That might ultimately get them off Cogent’s network, but it will have made little difference in the broader fight against DDoS.
His objective is to get the customer’s networking team to understand that they are the source of problematic spoofed traffic and address it. If they don’t, he can either impose strict BCP38 filtering from his side and risk blocking legitimate traffic or, as a last resort, refer them to his abuse team.
Using an email template that describes how spoofed traffic fuels DDoS attacks, Aaron reaches out to the customer with a shareable link to the view above in Kentik showing the spoofed traffic coming from the customer’s network.
Engaging with customers can be a very time-consuming process. There can be language barriers, network engineers who are either overworked or poorly trained, and, unfortunately, some networking teams who are simply uninterested in fixing the problem.
A certain cat at a cloud provider made a humorous anti-spoofing reflection response bingo card to capture the variety of unhelpful responses encountered when trying to bring spoofing to the attention of an external peer network.
Yielding results against DDoS
It is a painstaking process, but it is part of a larger effort that has begun to yield results. One of the key insights of those battling DDoS was that there was a finite amount of networks which were responsible for launching most attacks. This insight meant that if the large service providers coordinated with each other, they could identify and eliminate many of the sources of spoofed traffic and thus make a dent in the number and severity of DDoS attacks.
In a webinar I did earlier this year with backbone carrier Arelion’s Chief Evangelist Mattias Fridström, he reported seeing a decline in the number of DDoS attacks seen across their global network and chalked it up to this collaboration between Tier 1 operators (service providers). Mattias’s slide is below:
Another factor contributing to the reduction in DDoS is the work of the “Big Pipes” effort, profiled in WIRED back in May. This group is another example of experts from multiple (sometimes competing) companies working together with the common objective of eliminating DDoS attacks. Big Pipes works with law enforcement to take down DDoS-for-hire services, better known as booter services.
The bottom line is if your network is allowing spoofed traffic, there is a good chance that someone is taking advantage of this fact and using a connection on your infrastructure to launch DDoS attacks against victims around the world.
If you run a network that operates as a service provider, whether a university or a backbone carrier, you have a responsibility to the rest of the internet to actively look for and eliminate spoofed traffic. And a solution like Kentik is perfect for the job.
If someone like Aaron reaches out to you about spoofed traffic coming from your network, you’ll need tools in place to investigate and address the claims. You don’t want to be the one to complete his anti-spoofing response bingo card.
