Incident Response

By John Locke on July 27, 2011

All the planning and preparation in the world won't prevent an incident, but it can greatly reduce the consequences.
Nothing better prepares you for responding to disaster than experience. In the world of web applications, sometimes we act as firefighters, coming in to rescue the smoldering remains of a hacked site, a crashed server, or an unexpected traffic burst.
As recently as two weeks ago, we had a development server's hard drive become completely wrecked, a complete and total loss. But not our data -- we simply recovered the sites we were actively working on on a new server, as needed -- a 5 - 10 minute process for each one. (And they got an upgraded environment at the same time!) In another case, we restored an email for a client who had deleted it months earlier, potentially keeping her out of a law suit.
In one of our biggest incidents, a production server got hacked a couple years ago, through a malicious file upload in a Joomla site (which we can't secure using the technique I described for Drupal above). We detected the hack within a couple hours, and were easily able to block out the malicious code and close the previously unknown vulnerability. At that point, we could not easily tell if the attacker had left some other malicious code in place, so we decided to be prudent and retire that server. Within two days we had moved every production site over to a brand new server with known good code, with no downtime (other than for the site which had gotten hacked, which we kept offline until we had diagnosed and shut down how the attacker had broken in).
This kind of break-in happens all the time on regular shared hosts. If your web site isn't important to your business, then perhaps you can get away with a $10/month host. Do you have an adequate plan to cover all the likely ways your site can get destroyed? Can you ensure that anything that goes wrong with your web site will have minimal impact on your online reputation?
If your web site IS important to your business, and you don't have the time or staff to keep it secure, backed up, and well maintained, I encourage you to check out our support plans. For less than the cost of two hours of our time a month, we will maintain two sets of backups for your Drupal site, provide shared hosting on a much more secure environment, maintain a test copy of your site, perform security updates and maintenance, and prevent you from losing more than 1 day of data.
If you have higher security needs, such as PCI compliance (for accepting credit cards online) or protection of personally identifiable information (generally health care industry), need to have login access to the server for other developers, use huge amounts of disk space or bandwidth, or want to prevent even 1 day of data loss, we can most definitely help, but it will cost more.
Give us a call at 206-577-0540 or drop us a line, we'd love to help keep your online business presence safe!

Free Tagging Area

Add new comment

The content of this field is kept private and will not be shown publicly.

Filtered HTML

  • Web page addresses and email addresses turn into links automatically.
  • Allowed HTML tags: <a href hreflang> <em> <strong> <blockquote cite> <cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h1> <h2 id> <h3 id> <h4 id> <h5 id> <p> <br> <img src alt height width>
  • Lines and paragraphs break automatically.

Drupal Canvas — Block HTML (locked)

  • Allowed HTML tags: <strong> <em> <u> <a href> <p> <br> <ul> <ol> <li>

Drupal Canvas — Inline HTML (locked)

  • Allowed HTML tags: <strong> <em> <u> <a href>

More Like This

Grafana line showing load dropping to normal
🕑Aug 22, 2023 🖋John Locke 💬2

Rate Limiting an aggressive bot in Nginx

High load isn't necessarily an emergency, but it may be a heads-up before a site noticeably slows down. Sometimes there are weird spikes that just go away, but sometimes this is an indication of a Denial of Service.

Code monster
🕑Mar 29, 2018 🖋John Locke 💬3

Drupalgeddon2: Should I worry about critical security updates?

No, you should not. You should let us worry about them, and go back to your business.

Seriously, we're getting questions from all kinds of people about whether this matters. I'm a bit surprised that there is any question about that. Would you be concerned if your top salesperson was selling for somebody else? If your cashiers were jotting down credit card numbers when they charged a card? If your office became a well-known spot for illicit drug or gun dealers? If your office had a bunch of scammers squatting and running a pyramid scheme? If your confidential client information could be revealed as easily as using a bic pen on an old Kryptonite lock?

Bic Pen vs Kryptonite Lock

We've seen some variation of every single one of those scenarios. And all of them are possible with a remote code execution flaw in a web application, like yesterday's Drupal security vulnerability.

And yet people still

Meltdown
🕑Jan 15, 2018 🖋John Locke 💬0

Meltdown notes

The Meltdown vulnerability leaked out into public news a full week before patches were available for many distributions. When patches did become available, sometimes the patch caused further trouble.

Meltdown in action
🕑Jan 11, 2018 🖋John Locke 💬1

The Spectre of a Meltdown

The news was supposed to come out Tuesday, but it leaked early. Last week we learned about three variations of a new class of attacks on modern computing, before many vendors could release a patch -- and we come to find out that the root cause may be entirely unpatchable, and can only be fixed by buying new computers.

Today Microsoft released a patch -- which they had to quickly pull when they discovered that it crashed computers with AMD chips.

Essentially Spectre and Meltdown demonstrate a new way of attacking your smartphone, your laptop, your company's web server, your desktop, maybe even your tv and refrigerator.

Meltdown - Animated
Meltdown in Action

This all sounds dreadfully scary. And it is... but don't panic! Instead, read on to learn how this might affect you, your website, and what you can do to prevent bad things from getting worse.