DDoS Attack Thursday March 20th – Post Mortem

Written by: on March 27, 2014 Posted in:

Last Thursday at about 6:40 a.m. we received an email threatening a Distributed Denial of Service (DDoS) attack on our systems, unless we paid a ransom of 2.5 bitcoins. Within minutes an attack was launched and hootsuite.com became inaccesible. Our OnCall team immediately went to work to mitigate the attack.

In a DDoS attack, a large number of compromised computers (collectively called a botnet) repeatedly sends many simultaneous web requests to a website, which overwhelms the site’s ability to process regular traffic.

To a regular hootsuite.com customer, it appears like the site is unresponsive or in the best case really, really slow. Though such an attack is annoying and potentially costly to the customer, it’s important to note that in no way was any hootsuite.com customer data compromised. The attack was directed at the system that handles incoming web requests – our load balancer grid. Our databases and other internal systems were unaffected. In fact, during the attack scheduled messages continued to be sent.

After verifying the attack was legitimate, HootSuite’s OnCall team focused on getting our load balancer (LB) grid back online. We first attempted to identify and isolate the malicious traffic so that we could block it before it reached the LBs. However, our LBs were unable to perform diagnostic activity because they were maxed out handling incoming traffic. Next, we turned our attention to scaling up our capacity to handle the incoming traffic. Fortunately, we have a solid configuration management system in place powered by ansible, which allows us to spin up new LBs and add them to our grid quickly. That combined with the elasticity provided by AWS enabled us to ramp up the capacity of our LB grid to handle the traffic from the attack and process regular traffic at the same time – bringing hootsuite.com back online for our users at approximately 9:40am. We then continued to triage the malicious traffic in an attempt to block it, however about an hour after we brought the site back up, the attack stopped.

We recognize that HootSuite is an essential tool for our customers, and as such we invest a lot of effort in making sure it is available 24/7. We’re taking the following steps to reduce downtimes and improve resiliency to DDoS attacks in the future:

  1. Hardening the outer layer of our architecture so malicious traffic gets dropped instead of bogging down servers
  2. Improve effectiveness of our monitoring systems which will allow us to pinpoint the problem quickly
  3. Improve on our auto scaling infrastructure to make it even faster and easier to add capacity to handle attacks

You may be wondering why we didn’t just fork over the 2.5 Bitcoins (about $1200) to pay off the attacker. For one, we are not interested in negotiating with criminals. Additionally, paying the ransom would likely lead to future attacks.

It’s probable that the attacker was testing our willingness to bargain – 2.5 Bitcoins is high enough to sound legitimate, but low enough that there was a reasonable chance we would pay. If we handed over the money it’s likely another ransom would follow with an even higher price tag.

It’s important to us that our customers know what’s going on during an outage – we commit to posting an update to status.hootsuite.com as soon as we detect an issue and at least every 15 minutes after that until it’s resolved. The steps outlined above will ensure we can diagnose issues even faster and get updates out to customers as quickly as possible.

I would like to thank the folks at meetup.com who were eager to help and provided valuable information to our team, having suffered a similar attack a week before. Our team would be happy to collaborate with anyone suffering a similar attack – feel free to reach out to me on twitter at @sedsimon if you have information. In closing, I would like to offer my sincere apologies to all customers affected by this attack, and to thank you for your patience and continued support.

Why Using RequireJS As-Is Could Hurt Your Web Performance

Written by: on March 25, 2014 Posted in:

At HootSuite, we recently converted our entire JavaScript codebase to AMD modules that we load using RequireJS, the must-have “script loader that improves speed and code quality.”

After doing so, we quickly realized that following the guidelines indicated on RequireJS.org would have the opposite effect on our dashboard and make us lose the benefit of the biggest browser performance improvement ever: the look-ahead pre-parser. Continue…

How To Make Jenkins Play Nicely With Git

Written by: on March 19, 2014 Posted in:

At HootSuite, we use Jenkins as our continuous integration tool and Git as our source code manager (SCM), which we host in-house with GitLab.

Our git workflow consists of short-lived feature-branches that are merged into the production branch as soon as possible. We build and test each feature branch as it is pushed to the Git remote. This gives our engineers feedback before we merge to production and avoids delays in our continuous deployment pipeline.

Unfortunately, Jenkins is not suited to using Git with a feature branch workflow. Jenkin’s Git plugins assume a single branch build which obviously contrasts to using multiple feature branches.

This blog post will detail how we overcame these deficiencies to successfully use Jenkins with a multiple feature-branch Git workflow.

Continue…

Git Pull, Composer install

Written by: on February 26, 2014 Posted in:

If you work on a large team and you use Composer to manage your PHP Packages, sometimes Composer changes being introduced will cause everyone a little headache.

This Gist Git ‘post-update’ hook will check your file change list after a pull and do a composer install if necessary.

Git post-update hook to trigger Composer Install

A PHP Management Script, in Python. #bsu.

Redis > (memcached+MySQL) for Team Assignments

Written by: on February 12, 2014 Posted in:

Team assignments outgrew its data-store

What are Team Assignments?


Team Assignments are used by organizations to assign social messages to their team members. If a message comes through one of your social streams in HootSuite, you can assign it to a particular user, or team, to respond to. Details on how we did it after the fold.

Continue…

Upgrading Our Publisher Series: Part 2 – MongoPHP Driver Update

Written by: on October 8, 2013 Posted in:

TL;DR

  • Create a timeline for the day so people know what to expect
  • Set a final commit time for everyone that needs to get code to production on upgrade day
  • Jenkins helps control what code makes it out to production
  • Vagrant ensures consistent dev environments
  • Ansible makes it easy to provision your servers
  • Set a clear point of no return. If you’re not 100% comfortable with the way things are going, cut out early and try another day

Continue…

Upgrading our Publisher Series

Written by: on September 26, 2013 Posted in:

Part 1 – Mongo Message Migration

Here at HootSuite, our members rely on our Publisher to send and schedule messages to their social networks.

Due to a tremendous amount of growth over a very short period of time, our Publisher has been through some growing pains and we decided it was time to move our Publisher data to its own infrastructure.

Continue…

Functional testing with CasperJS

Written by: on August 21, 2013 Posted in:

The HootSuite web dashboard is quite front-end heavy. We have a large JavaScript codebase and a lot of it was written using jQuery, with most of the code touching the DOM in some way. Due to the coupling of the code with the DOM, unit testing gets a bit complicated as HTML fixtures will have to be supplied for tests. It’s not impossible, but it’s a bit more work, and experience shows that developers are more likely to write tests when they’re convienent. We’ve adopted to use CasperJS to fill the gaps in our unit test coverage.

Continue…

Migrating ow.ly

Written by: on July 17, 2013 Posted in:

This is an adaption of a talk I gave at our weekly departmental Lightning Talks. This is the story of how we migrated our link shortener Ow.ly to new infrastructure.

Once upon a time, ow.ly was hosted on a single instance, and all was well. At least that’s what I have been told — it’s a story lost to the mists of startup history and engineers reminiscing over drinks.

Continue…