At Freelock, we don't think one backup is enough. All kinds of things can, and often do go wrong. Murphy was an optimist, after all.
We make sure that all of our code and customers' data is backed up not just once but multiple times, in multiple places. Perhaps we go overboard with this, but over the course of nearly a decade of business, we have not lost more than a few hours of data for any of our customers we cover. And we've recovered data from hardware failures, hacked servers, accidental deletion, and more. Our backup systems have not just been tested, but they have successfully been used to recover failed systems. We have years of experience helping customers prevent and recover from all sorts of failures.
We make sure there are at least two backups of everything we do. For production sites, we keep an image backup of each server at the host where it can be spun up in a matter of minutes on brand new hardware, and a file-based backup of everything on the server on our backup server in our office. For development sites, we have the same file-based backup server in house, and we mirror all our data out to a dedicated backup service on the Internet -- if our office burned down, servers along with it, it would only take a few days to restore. Our customers would not lose any data or have to start over.
We have a sophisticated backup tool that allows us to cram huge amounts of data into relatively small disk space. By using compression and a sophisticated de-duplication algorithm, we're able to store some 3 terabytes of data in around 300 gigabytes of disk space -- and keep particular snapshots of every file on every server for up to 16 months. Without a historical backup, you can be screwed if you accidentally delete something or if an attacker breaks in and you don't notice for a few days.
In addition to all of these backups, often our developers will have local copies of the sites they are working on, with the full project history. We get this by using the same code management tools that Linux itself uses, storing every change in the entire project on every instance of the site.
Backups are just the safety net, though. They help you when you have a problem. Some problems are unavoidable, so you cannot do without backups -- but many can be avoided fairly easy. Read on to find out how.
Backups
More Like This
We Patched wp2shell in 4 Hours. On a Friday Afternoon.
On July 17, WordPress shipped an emergency security release for a vulnerability chain now being called wp2shell.
Tempting Fate: What Happened When One of Our "Protected" Sites Got Hacked
The Rules Have Changed: Security in the Age of AI-Assisted Attacks
Rate Limiting an aggressive bot in Nginx
High load isn't necessarily an emergency, but it may be a heads-up before a site noticeably slows down. Sometimes there are weird spikes that just go away, but sometimes this is an indication of a Denial of Service.
This is Why You Need Another Backup
At 12:27pm, our alerts started firing. Multiple ones -- website down, server down, secondary monitoring -- one of our client's servers had completely disappeared off the Internet.
Drupalgeddon2: Should I worry about critical security updates?
No, you should not. You should let us worry about them, and go back to your business.
Seriously, we're getting questions from all kinds of people about whether this matters. I'm a bit surprised that there is any question about that. Would you be concerned if your top salesperson was selling for somebody else? If your cashiers were jotting down credit card numbers when they charged a card? If your office became a well-known spot for illicit drug or gun dealers? If your office had a bunch of scammers squatting and running a pyramid scheme? If your confidential client information could be revealed as easily as using a bic pen on an old Kryptonite lock?
We've seen some variation of every single one of those scenarios. And all of them are possible with a remote code execution flaw in a web application, like yesterday's Drupal security vulnerability.
And yet people still
Meltdown notes
The Meltdown vulnerability leaked out into public news a full week before patches were available for many distributions. When patches did become available, sometimes the patch caused further trouble.
Add new comment