At 12:27pm, our alerts started firing. Multiple ones -- website down, server down, secondary monitoring -- one of our client's servers had completely disappeared off the Internet.
I confirmed that I could not reach the site, or the server, and then looked up our AWS credentials for that customer. They didn't work. Then I tried to call two different contacts at our client, leaving messages, and sent an email asking if they needed assistance restoring their AWS account.
The answer came back 20 minutes later, after our client checked their email account associated with AWS.
Subject: Irregular activity in your AWS account: xxx
Severity: Urgent
Correspondence: Dear AWS Customer,
Your AWS Account may be compromised! Please review the following notice and take immediate action to secure your account.
It turns out that another vendor had posted an AWS API key to a public Github server! This is like putting your password out on the Internet for anybody to see and use.
Worst Case Scenario
A malicious attacker, with an API key granting sufficient access to an AWS account, can shut down and delete your servers, your other accounts, your storage volumes. Worse, they could easily delete your backups, if your backups are managed in the same account.
This is why you should always have backups somewhere else entirely.
In our client's case, the worst case would not have been business-threatening -- we had rolled out an update to their site just last Friday, and our deployment system automatically takes a backup snapshot before each release. So we had all their code, and a relatively recent database. We did not have all of their assets (uploaded images, photos, etc) so there would be some loss, but we had some from months ago, and this is not an asset-heavy site, so they could have recovered.
But if they had taken advice we had given them just a couple months ago and set up a secondary backup, we could have restored them with only a few hours of missing data.
A stroke of luck
It turns out the other vendor had taken the site down. They had gotten the notice about the published key, went into the account and shut off every server they didn't recognize, disabled (or in our case, deleted) the IAM accounts they didn't recognize. And taken out the public website as a result.
Fortunately, the only thing deleted was our IAM account. This was easy enough to recreate, and everything else we were able to turn back on. Other than a little over an hour of downtime, and some emergency service, they escaped unscathed.
Are all your eggs in one basket?
If you don't have a full backup at an entirely different location, using an entirely different provider, you may have no backup at all if an attacker gains access to your hosting credentials.
If you don't have historical backups, or nightly checks on the integrity of your website and assets, you could lose data through corruption or infection, and if you don't notice before your backups rotate through to deletion, your backup strategy is not robust enough for the real world.
If you're not sure, we can answer that for you. Our Site Appraisal with a Security Assessment will answer these questions, and more! Give us a call and we'll get you squared away.
Add new comment