Rapid real-world testing using git-deploy

Ævar Arnfjörð Bjarmason <avarab@gmail.com>
FOSDEM – February 5, 2012

What is git-deploy?

• A deployment tool conceived and introduced at Booking.com in 2008
• We've made it open source, clone it on GitHub.
• You should use it, and here's why...

It's a tool to make deployments really easy

We've gone from deployments being a hard manual process, to doing dozens of them a day.

It's highly configurable and pluggable

• We use it to deploy around 20 environments.
• Those 20 environments have around 60 hooks between them to customize deployments.

What's the core concept behind it?

• git-deploy will create an annotated git tag for all your rollouts.
• This gives you permanent git-level history of all your rollouts..

So how does it actually work

• Rollouts are done from a designated git repository cloned from your main repository.
• Only one user can do a rollout at a time.
• Once you've locked the deployment server you can perform any git operation you want (git pull/commit/cherry-pick/reset).
• When you're ready you sync. This creates a tag.
• This calls your sync hook.
• When that's done you finish, which unlocks deployments, pushes the tags, sends E-Mail etc.

Step 1: start

• Creates a .git/deploy/lock file in your deployment repository.
• Does a git pull for you.
• Runs any start hooks.

  # checks that we're on "master"
  cron/pre-start.001_nobranch_rollout.pl
  

• You can also do hotfix, it's just a start without the git pull and the start hooks.
• You can always abort the deploy at this point before you sync.

Step 1: start - example of someone already deploying

• $ git-deploy start
  # INFO : Starting step 'start' at 2012-02-02 16-22-01
  # FATAL: You may not start a new rollout as it looks like one is already in progress!
  # FATAL: Failed to create lock dir '.git/deploy' because 'File exists'
  # FATAL: Log:
  # FATAL:        start:  2012-02-02 16:19:28     trunk   110bc23d8204dae87b2af259a02d47cd08711589        717     slanning
  

Step 1: start - example of a successful start

• Output truncated, runs staging webserver restart hooks and has more useful output:

  $ git-deploy start
  # INFO : Starting step 'start' at 2012-02-02 16-30-12
  # INFO : Working directory was checked out to tag 'app-20120202-160118', will restore this tag on abort
  # INFO : Step 'start' concluded at 2012-02-02 16:30:12 took 29 seconds to complete
  # YAY  : 'start' for 'app' completed successfully
              

• Now ready for the sync step:

  $ git-deploy status
  # NOTE : app rollout started - not synced yet (app-20120202-160118)
  # NOTE : start: 2012-02-02 16:30:12     trunk   639022c78daf1b85b76d4816803a6d37bef67e10        3374    aearnfjord
              

Step 2: sync

• This is when you've decided that what you have staged is good enough.
• It creates a tag to permanently tag this commit as having been rolled out in this environment at this time.
• The tag will be: prefix-YYYYMMDD-HHMMSS
• Calls a sync hook that you define.
• If your sync hook has an error at this point it's up to you to resolve it.

Step 3: finish

• Run for you when sync finishes successfully, or manually if your sync hook had an error.
• Pushes your shiny new tag to your main repository.
• Sends out an E-Mail announcing your new rollout if you've configured it to do so.
• Unlocks the staging server so you can do another rollout.

Step 4: panic

Why use it?

• It makes reverts really easy. Helpful when you've just broken something.
• You can see your your entire deployment history at the git level.
• It does really exhaustive error checking -- added over years of trial and error as its inexperienced users have tried to screw up rollouts in every way possible.
• It ensures that the appropriate tags are created, and that they're pushed.
• Extensive puppet-friendly configuration file using git-config(1)
• Extensive hook support for hooking into every step of deployments.

What doesn't it do?

• Worry about how your code is synced out, that's your job.
• git push
• rsync it.
• make a Debian or RPM package out of it.
• tar it up, gpg sign it and mail it around.
• strap it to a pigeon.
• This means you can most likely use it with your existing deployment infrastructure.

Why does it work for us / Why do we need it?

• We do Deployment-Driven-Development

  

• Who cares if it passes the tests? Do customers like it?
• Which means less emphasis on unit testing.
• Which means more frequent deploys to see if you're still OK.
• ...and rapid rollbacks if you're not.
• Enough monitoring can replace testing, but testing can never replace monitoring.

Thank you!, questions?

• Source and documentation: http://github.com/git-deploy
• This talk is available online at git-deploy.github.com
• Contact me at github.com/avar