Our first DDoS attack!!

Yesterday biicode suffered a DDoS attack on this wordpress blog.

DDoS attack (Distributed Denial of Service) tries to make a resource (this blog) unavailable using servers distributed all over the world.
In this case, sources came from Korea, USA, Europe… More than a hundred simultaneous connections brought down our blog.

 

DDoS attack can seriously damage your system.

DDoS attack

What happened?

We received an alert about our blog not responding right. We connected inmediatly to the server and started looking into the server logs.

Why did the blog go down?

Well, the first sign that something was going wrong was a message telling us the server couldn’t connect to the database.
At that moment we connected to the server and tried to restart mysql service… but the service went down almost inmediatly.
We looked at the mysql error log and we found it was a memory issue.
After looking up in google the error code we decided to increase the server memory (easy to do with a virtualized instance) but it didn’t work, memory wasn’t being used and the problem persisted.
We even tried to increase the swap partition… but our disk got full.

What the f*** is going on!!!??

That was our reaction before all we tried failed.
We started to think that maybe we were under some kind of attack.
We executed netstat command to look at port 80. More than a hundred connections were stablished.

With these command options I could check the connected IPs:

Well… we definitely are under a DDoS attack…now what?

Repealing a DDoS attack may be very complex. The most important thing is to understand the attack and think of a way to stop it.
We looked in the Apache access log and we saw hundreds of access to the same resource: “xmlrpc.php”

Xmlrpc.php is an API to remote publish contents to wordpress. Attackers are POSTing big files to this resource. Files were not getting published though, because the attacker didn’t have the right credentials, however, WordPress was receiving a laaaaaaarge amount of data and loading it into server memory which eventually was full.

The solution was simple… We moved that php file to a non accessible location. The server started to respond 404 for all these requeste and all the system went to normal. We don’t need xmlrpc and after what happened… We prefer never to use it again.

Conclusion

Despite this, we are happy with the results of this DDoS attack. We are in beta, we are not perfect, our systems are not configured perfectly, but we are learning a lot and we are greatful to be able to fix any problem that affects our fantastic users!. Also we know that we are gaining interest and an attack like this confirms that!

Happy summer!!

Stay tuned


Related Posts
  • sajithkumar kanangottu

    even I had a DDos attack yesterday. the attacker selected index.php file. So i could not do what you did. Finally I have installed firewall, that blocked all 200 zombies. I have written the steps I have taken to prevent DDos attack in future. May be this will help you also

    http://sajith.snydle.com/what-is-ddos-and-how-to-stop-ddos-attacks.html

    Regards
    Sajith