title: DDOS mystery involving Linux and mod_ssl
date: '2009-10-12T01:53:03+00:00'
permalink: ddos_mystery_involving_linux_and
layout: post

<p>In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at <a href="https://issues.apache.org" title="https://issues.apache.org/">https://issues.apache.org/</a>.&nbsp; On further inspection we noticed a huge amount of &quot;Browser disconnect&quot; errors in the error log right at the beginning of the ssl transaction, on the order of 50 connections / second.&nbsp; This was grinding the machine to a standstill, so we wrote a quick and dirty <a href="http://people.apache.org/~joes/ddos_accept.pl">perl script</a> to investigate the matter.&nbsp; Initial reports indicated a ddos attack from nearly 100K machines targeting Apache + mod_ssl's accept loop, and the script was tweaked to filter out that traffic before proxying the connections to httpd.</p><p>As we started getting a picture of the IP space conducting the attack, the prognosis looked rather bleak: more and more IP's were getting involved and the ddos traffic continued to increase, getting to the point where Linux was shutting down the ethernet interface.&nbsp; So we then rerouted the traffic to an available FreeBSD machine, which did a stellar job of filtering out the traffic at the kernel level.&nbsp; We unfortunately didn't quite realize how good a job FreeBSD was doing, and for a time we were operating under the impression that the ddos was ending.&nbsp; So we eventually moved the traffic back to brutus, the original Linux host, and <a href="http://people.apache.org/~joes/avoid_dos_2.2.x-try2.diff">patched httpd</a> using code developed by Ruediger Pluem.<br /></p><p>And back came the ddos traffic.&nbsp; In a few days the rate of closed connections had nearly doubled, so we had little choice but to start dumping the most frequent IP addresses into iptables DROP rules.&nbsp; 5000 rules cut the traffic by 2/3 in an instant.&nbsp; But the problem was growing- our logs indicated there were now over 300K addresses participating in the attack.</p><p>We started looking closer at the IP's in an attempt to correlate them with regular http requests.&nbsp;&nbsp; The only pattern that seemed to emerge was that many of the IP's in question we're also generating spartan&nbsp; &quot;GET / HTTP/1.1&quot; requests with a single Host: <a href="http://140.211.11.140">140.211.11.140</a> header to port 443.&nbsp;&nbsp; Backtracking through a year of logs revealed that these spartan requests had been going on since August 6, 2008.&nbsp; The IP's originating these requests were as varied as, and more often that not matched up with, the rapid closed connection traffic we started seeing in October.<br /></p><p>So what exactly is going on here?&nbsp; The closed connection traffic continues to rise, and the origin of the associated spartan requests is currently unknown.</p>
