The other night as I was getting ready to sleep, I received an email from the host of my personal Linux VPS saying that I had exceeded my monthly transfer quota. I didn’t pay much mind to the warning, as the excess transfer was insignificant, and at that time I was too tired to care. I closed my email, got into bed and fell asleep.
The next morning I woke to find several more messages from my VPS host, each with a higher and more significant excess transfer than the last. At this point it occurred to me that it was unusual for my VPS to reach its quota, let alone exceed it. The excess transfer was now enough that it was going to incur significant cost, so I set about investigating the cause.
I downloaded some firewall logs for the previous few days from the server and imported them into Vantage. The first place I looked was in an analysis at the “Source Address” summary, to see where the activity was coming from. What I found was a single host with a disproportionately larger amount of transferred data than the other addresses listed, so I drilled down to the “Destination Port” summary for this source address to see what services it was accessing. I found that all the traffic was going to port 53 – my DNS. More accurately, the large amount of data was going from my DNS to the source address. Drilling down to the “Individual records” view then showed that my server was providing a large response to a small DNS request from that source address – about 20 times per second.
Curious about why this single machine somewhere on the Internet was bombarding my server with small DNS requests at such a high rate, I set my server’s firewall to deny packets from that address and began searching around online for any information.
I quickly found out that I hadn’t configured my DNS properly, and it was set to allow recursive requests, meaning that if a request came in for a domain my server wasn’t authoritative for, it would then forward the request to another DNS that could answer, or given a blank request it would respond with the full list of root servers. Running tcpdump on the VPS revealed that every request coming in was blank, and my server was responding with the full list of root servers for each request.
It still seemed odd that a server would be constantly sending small requests to my server and receiving large responses. Then it dawned on me; I was looking at a Distributed Reflected Denial of Service (DRDoS) attack. The source address in all the requests I had looked at was forged by the attackers, so that my server – and many other servers out there also receiving the requests – would send their responses to the forged source address in an attempt to flood its connection. The source address in my firewall logs was the target of the attack. I found more information about this specific type of attack here.
Having disabled recursion on my DNS, my server’s contribution to the attack was significantly reduced. However, my server was now responding with a much smaller “request denied” packet for each incoming request. I wanted some way of preventing my DNS from responding at all, so again I headed out to the Internet to see what I could find.
I discovered a package called “fail2ban”, which dynamically updates your firewall rules to block addresses that are abusing your server’s services. I installed it using this guide, and immediately my bandwidth usage dropped off as it blocked further DNS requests. Even now the requests are still flooding in, but now my VPS contributes only a handful of packets towards the attack instead of the previous millions per day.
Leave A Comment