More and more I keep running into assertions that Git is a version control tool, and that if you use it for deployment, you're doing it wrong.
At Freelock we find it to be a very effective deployment tool, and I'm not seeing a solution that meets our needs any better.
Two presentations in particular caught my attention recently mentioned this:
- In the Modules Unraveled Dropfort podcast, @matthewwinstone has built infrastructure around deployment using Drush make files
- At a Drupalcon session on using Composer with Drupal 7
... but those are just two that come to mind. I feel like I've heard this message from lots of different directions, particularly among some of the broader PHP community.
But again, why? What is wrong with using git as a deployment tool?
Reasons to not use git as a deployment tool
As best I can tell, there are three main arguments against using git for deployment. These boil down to:
- Storing environment-specific information, including security risks around database credentials, API keys, etc.
- Storing object files, generated files, code unrelated to the development of the project.
- Disk usage, wasted space taken up by having the full history of a project in a production deployment.
I think the main thrust of this line of thinking is that "git is for source code management, you don't need that in delivered production code, and storing all of this other binary/environment stuff gets in the way of developing the upstream code for a project."
When we started using git for deployment, we ran into these issues, and lots of challenges particularly around managing multiple environments. But they are entirely solvable, and git brings with it a number of big positives as one part of a deployment toolchain.
Why I think git is a great deployment tool
Git brings several big benefits to deployment:
- It's extremely fast, can merge in a huge number of changes in a few seconds.
- It's very fast and easy to roll back changes that go awry. Especially if you use tags effectively.
- For managing lots of related websites, it's an extremely effective way to provide patches across a bunch of different sites.
- With a cryptographic hash of every code file, it's extremly simple to detect unauthorized modifications of a file -- e.g. detect if you've been hacked.
Git does not provide these benefits just by using it. For it to be an effective deployment tool, you need to have a very clear organization, relatively strict control over your processes, and some guiding principles. But with those in place, I haven't seen any tool that does a better job for deployment than git.
How we use git for deployment
We've spent years refining our git deployment process, and it actually feels very dialed in at this point. It's not the only tool we use -- we use a bunch of other tools and scripts to help automate and streamline this process, but here are a few of the ways we've addressed the problems with using git for deployment, and leveraged its benefits:
Keep environment-specific settings out of git
Acquia does this. So does Pantheon. Most hosts that rely on git either provide their own environments entirely, or leverage some sort of include structure.
We commit the settings.php file, and can add $conf variables there that need to be available in all environments. At the bottom of the settings.php file, we use a PHP include statement to include a local settings file that is excluded from git, and in that file we store database credentials, environment indicator settings, API keys (for production), enable/disable variables for things like securepages and reroute_email, and anything else that only applies to a single environment.
We catalog environments in drush alias files, one for each client site. These aliases include modules to enable/disable whenever a database sync is performed into an appropriate environment (e.g. into staging, sanitize emails, enable dev modules, disable payment modules, etc).
We distribute drush alias files using Salt Stack, a configuration management tool similar to Puppet, Chef, or Ansible. We're working on moving more environment-specific settings (mainly for provisioning Docker containers or kicking off test scripts) into Salt.
Add generated code to git
You'll find all sorts of guidance to exclude vendor code, things that can be pulled down with make files, things generated with Sass/Compass/other tools out of git.
Coming from an operations perspective, why on earth would you want build tools present on a production server? A huge amount of setting up tools like Compass and Composer can put you into dependency hell, even with composer.lock and gem.lock files. It seems Docker is what's in vogue these days for managing that -- run a bunch of build scripts in a Docker container, and then copy that up to production -- except that nobody who does that dares to run these Docker containers in production. And you've added a bunch of steps to getting something you can actually run on a server, requiring re-compiling and rebuilding everything for the tiniest change.
Git handles this way, way better. And for any scenario where you need tools to build something that you're actually delivering, why not commit that working CSS or whatever you're generating, into your build and pushing that out? No interim container builds necessary. No risky build tools lying around on your production server. No downtime waiting for builds to run on production -- just deploy and clear some caches.
Buy more disk space and grab a coffee
Having a copy of every single Drupal core commit ever in every single production copy may be unnecessary. It does make a clone of the git tree take a few minutes, and take up a couple hundred megabytes of disk.
While that doesn't really add any significant value to a deployment process, and while there's really no benefit to keeping all that around in so many copies, we've found that it's just not worth cleaning all that out.
First of all, these penalties really only make any difference whatsoever the very first time you create a new copy of the site. After that, updates are really really fast.
But the bigger reasons are all related to our core business: keeping dozens of Drupal sites up-to-date.
We've written about our branching and updating strategy before. In short, we maintain our own cloned copy of the main Drupal.org git repository, with branches for Drupal 6, 7, and Pressflow. Upgrading to a new point release of Drupal means merging the new tag into our branch. Then we've written a script to update all the contrib modules we curate across most of our sites. We apply patches to both Drupal core and contrib (for example, patching this core issue, and many changing patches for Media 2.x and other modules).
We call this clone our "upstream" and from there we can simply merge these updates into all of our client's dev sites, run update scripts, and test.
Trying to remove or squash the commit history of Drupal always seems to cause a lot more problems in this deployment process than it solves, so we just live with the extra disk usage and go refill the coffee cup.
Git is a "distributed version control system"
... This seems to be the biggest point that these "purists" who think git should only be used for version control miss: it's extremely cheap and easy to clone and maintain different branches with git. That means you can keep your upstream pristine clone for developing a project that you want to publish on Drupal.org, and very easily use a different git clone to manage it for deployment in a particular site. There is no reason you can't do both with git.
On to the stuff we leverage...
Git post-update hooks for automated deployment
We use a central git server running gitolite. In our gitolite server, we've added a post-update hook that looks for commits pushed that have a branch name matching "release/*" or "hotfix/*". If such a branch is pushed, it passes the repository name, branch name, user who pushed it, and a couple other details into Jenkins.
Jenkins runs a shell script that makes heavy use of drush to connect to the site's stage copy, and automatically deploy there. If the branch already exists on the stage site, it skips the database copy -- otherwise it imports a fresh copy of the production database, runs through some sanitization, checks out the new code, runs updates, reverts features, and notifies us in chat when it's done. Then we can review the Jenkins log and the state of the stage instance to determine if there are manual steps we've overlooked, how long the deployment will take (e.g. to run update scripts, apply features), and whether or not we should plan to take the production site offline for the upgrade (we lean towards keeping it online unless there will be many minutes of entirely bad user experience...)
Production tags and deployment
We don't automatically deploy to production, but we have written scripts to support our process. Our stage cleanup script merges the current release code to master and develop branches, cleans up the central repository, leaves the stage copy on master and the development copy on develop. And it created a git tag for the particular release.
On production, we have nightly backups of the database, and the ability to easily take ad-hoc database snapshots. Before deploying we do a few quick checks:
- Is the database snapshot current enough? Take a new one if there are any update scripts.
- git status - Is there any uncommitted code? (Might indicate a bigger problem).
- drush fl - any features that are not up-to-date? If so, might need to update/pull back to development and re-roll the release.
- git log/git tag -- identify the current state of the code base -- should already be tagged with the previous release version. If not, create a new tag so we have an easy restore point.
Once ready, we simply pull down the new tag from master, run the updates, apply the features, and then follow any manual deployment steps that need to be done.
Roll back when hell breaks loose
With git, deployment of new code is fast. Once you've fetched the code updates, the merge process only changes files that have changed in the new release, and this process takes fractions of a second for small updates, a few seconds for large ones.
Rolling back is just as fast. Find a bunch of substantial problems? Simply git checkout release-x-1. Find some individual problem in a particular module? git checkout <myoldrelease> path/to/module, and then you can use other git tools to bring that back to development for a proper fix.
What other tool solves these deployment challenges?
Rolling back seems to be the most critical, and that's something you want to do immediately.
How can you do this with other tools?
What other tools are out there that can help with these challenges?
The main ones I hear about: Docker, build scripts/make files, alternative images.
We've actually started using Docker in a pretty big way, but not for managing deployments of site code -- git does a far better job for that, as we've already seen. At least if you're managing a bunch of different but similar sites -- if you are trying to deploy exactly the same site to a bunch of different servers to handle high loads, Docker may well have some advantages.
But the main way to roll back with Docker is to stop the container running the new site, destroy it, then run a new container based on the old image. You end up with some downtime, and if you're using Docker to isolate processes, you now need all the other linked containers to get connected to the new one, possibly triggering a cascade of Docker restarts and far more downtime.
Build Scripts/Make Files
What, are you crazy? Since your site isn't in git, you now need to go back to your development machine, check out an old copy of your site with the old versions of the make files -- if those had any version tags to begin with. If not, now you've got to figure out what version to go back to to get to a working state.
Utter madness. You might as well go back to your previous night's backup, but now you risk losing a bunch of data, too!
Alternative production images
I don't know of anyone using this deployment strategy with Drupal. But while doing some consulting for Microsoft, I ran into this pattern as the preferred way of releasing on Azure. With this pattern, you have not one but two copies of your production site, with essentially a load balancer in front of them. You deploy to the offline one, get it all up-to-date, and then switch all traffic to it. If anything goes wrong, you simply switch back to the other one.
Database schema changes were similarly captured as they were made, using tools that could reverse those schema/data changes if necessary.
Now that sounds like a very intriguing deployment strategy, especially the database schema management. But that's one tool the Drupal community does not (to my knowledge) have at its fingertips... yet.
And even so, the git deployment strategy we use complements this approach very well -- you still need to get the code onto the production instances somehow, and git seems far better than FTP/SFTP... and all the build-script/make file security issues still apply.
What's your take?
Our approach to deployment comes from a Dev-Ops, risk management point of view. We've developed our practices through thousands of releases on hundreds of different sites, with no major issues. We strive to make our production servers extremely secure, easily recoverable from bad releases, and running with as little downtime as possible. From this perspective, I'm not seeing any viable alternative to git for deployment that does as good a job...
What do you use for deployment? Why shouldn't we use git this way? Is there some magical deployment tool we're missing from our arsenal?