The rain had been falling on the city for weeks.
Not real rain. The kind that falls on the internet — a constant drumbeat of probes, scans, and automated fists rattling every doorknob on every block, every hour of the day. Most people don't hear it. That's fine. That's what we're here for.
My name doesn't matter. Call me the op. I run a small shop — we keep websites alive, patch the holes before the wrong people find them, and make sure that when something goes sideways, there's always a way back. It's not glamorous work. But this spring? This spring was something else.
The Case Files Start Stacking Up
It started, like most things do, with a tip.
A researcher — one of the good ones, the kind who finds a skeleton key and hands it to the locksmith instead of selling it to a fence — ran an AI scanner across a codebase. The codebase was NGINX (pronounced "Engine X"). You know NGINX even if you don't know you know it: it's the silent doorman standing in front of roughly a third of every website on the internet, checking credentials, waving traffic through. Reliable. Professional. Unobtrusive.
The scanner found a flaw that had been hiding in the doorman's coat since 2008. Eighteen years. Every human auditor who'd ever looked at that code had walked right past it. The machine found it in under an hour.
The flaw was bad. No password needed, no prior access, no special conditions. One crafted message to the doorman and you could put him on the floor. The patch went out. Then, within days — days — the exploits followed. That's the new clock. That's the world we're operating in.
We got our clients patched the next morning. But the doorbell was already ringing on the next case.
The Privilege Boys
You want to know what keeps an op up at night? Not the sophisticated stuff. The sophisticated stuff, you plan for. What keeps you up is the simple stuff — the bug so basic, so clean, so obvious in retrospect, that you can't believe nobody caught it before.
Meet Copy Fail. A logic flaw in the Linux kernel — the engine running underneath nearly every server on the internet. Any user on any affected system, no special skills required, could run a 732-byte script and walk out the other side with the keys to the whole building. No lockpicks. No crowbar. Just a very short note slipped under the door. It had been working since 2017 and nobody knew.
An AI found it. In about an hour.
We blocked the door before anyone could try the handle. But then — and this is the part where the city really shows its teeth — while we were still boarding up that window, two more opened.
The same week: Dirty Frag. Same family of bugs, different part of the kernel. Then Copy Fail 2: Electric Boogaloo — yes, that's actually what they called it — same underlying vulnerability, different angle of attack. Then Fragnesia, a follow-up that tried the same door with a different knock. Four separate privilege-escalation vulnerabilities in the same corner of the Linux kernel, inside of two weeks.
The city wasn't under siege. The city was the siege.
We ran the countermeasures across our entire fleet — blocking the kernel modules these attacks relied on, on every server we touch, before the upstream patches had even shipped. One of our clients runs an active VPN, so we threaded the needle there: partial block, kept the lights on, closed the threat. You learn the geography of a city by working in it long enough.
The Supply Chain Job
Here's a thing about the old days: you knew who the criminals were. They came in through the front door or they didn't come in at all. Then someone figured out a different play. Don't break into the building. Get hired by the company that stocks the vending machines.
The crew that runs under the name Mini Shai-Hulud has been working this angle since at least last September. They don't attack websites. They attack the tools that developers use to build websites. The packages. The libraries. The little pieces of trusted code that ship inside a thousand different projects because someone wrote them once and everyone else said good enough, use that.
In late April and again in mid-May, they hit npm, PyPI, and Packagist — the warehouses where developers pick up their supplies — simultaneously. Over 170 packages compromised. Libraries used by AI companies, enterprise software firms, and open-source projects with millions of weekly downloads. The payload was a credential thief: it walked out with GitHub tokens, cloud keys, CI/CD secrets, anything it could carry, then locked the back door behind it and waited.
Then came the hit that made everyone sit up straight: the Bitwarden job.
Bitwarden is the password manager. Not a password manager — the one that a lot of development shops trust with their keys to the kingdom. For ninety minutes on April 22nd, the Bitwarden command-line tool on npm was replaced with a weaponized copy. It looked identical. It installed identically. It just also quietly walked out with everything it could reach. It self-propagated — modified whatever other packages you had permission to publish, and re-uploaded them to spread the infection downstream.
Bitwarden caught it fast. User passwords weren't touched. But developer credentials, CI tokens, cloud keys? If you ran that install in that ninety-minute window, you had a problem you didn't know about yet.
The city's supply lines had been hit. And we were still watching the skies for what was coming Wednesday.
The Tip from the Drupal Gang
Word came down on a Monday.
The Drupal Security Team — and I want you to understand that these are serious people, careful people, people who do this right — put out a pre-announcement. Not the vulnerability itself. Just a warning: something big is coming Wednesday. Clear your calendar. Be ready to move fast.
They do it this way on purpose. While other projects have their patch work discovered in public repositories — giving the wrong people a head start on the exploit — the Drupal security team develops their fixes in a private repository, under strict embargo, until the moment they're ready to release everything at once. Theo from t3.gg called this out just days ago as exactly what every serious open-source project should be doing. Drupal's been doing it for years. That's not an accident. That's craft.
We spun up extra capacity. Verified every backup. Briefed the team. The hotfix pipeline was warm and ready — our system that skips the full test suite in an emergency, goes straight to deployment, but always checks the parachute is packed first. We'd be ready to have every managed site patched within four hours of the release dropping.
Wednesday came.
The Complication
The envelope opened. The threat was a SQL injection vulnerability — the kind that lets an attacker pull data straight out of a database through the website's front door. Serious. Nasty. The sort of thing that ends up in the news.
But.
It only affected Drupal sites running on PostgreSQL.
I looked at our client list. Every single one: MariaDB. MySQL. Not a PostgreSQL installation in the stack. The bullet had their names on it, and it missed every one of them by a matter of database choice.
I let myself breathe for approximately four seconds.
Then I read the rest of the security notes.
Nineteen Symfony vulnerabilities. Symfony is the framework Drupal is built on — the bones of the operation, the structural steel behind the walls. And tucked in among the nineteen: a critical Twig remote code execution vulnerability. Twig is the templating engine, the part that takes your content and turns it into web pages. Remote code execution means exactly what it sounds like: someone else's code, running on your server, doing whatever they please.
The SQL injection had been the headline. The Twig vulnerability was the real story.
Classic misdirection. Look at the front door while the crew comes in through the kitchen.
Clean Streets
Here's the thing about being ready for an emergency that turns out not to be an emergency: you're still ready.
The Twig and Symfony vulnerabilities were serious, but they weren't the kind of fire that requires skipping the safety checks. We had time to do it right. So instead of the hotfix process — fast, direct, no tests — we ran the full release cycle. Every managed site. Tests passing, Symfony patched, Twig locked down. All of them, same day.
Our clients went to sleep Wednesday night on fully patched sites. Not because we got lucky. Because we were ready before we knew what we were being ready for.
That's the job.
The City Keeps Moving
The rain's still falling. It always is. There are more cases on the desk — there are always more cases on the desk. But right now, tonight, every site we manage is locked up tight, backed up in at least two places that no single stolen key can reach, running software that's current as of yesterday.
If you're running a Drupal or WordPress site and you're not sure whether your shop has that kind of coverage — whether someone's watching your door, whether your backups are actually isolated, whether the next critical Wednesday will find you patched by Thursday — that's exactly what our Protection Plans are built for.
You don't have to know the city like we do. You just have to know who to call.
Get in touch. We'll take it from here.
Want the full technical breakdown of everything mentioned in this story? Read our detailed security operations post — every threat, every CVE, every countermeasure, no fiction required.
Add new comment