Recently, we’ve started to migrate from set deploy slots run by the DevOps team to an on-demand deployment system used by developers. The system we’ve set up is a push button based website, but has left us several interesting problems:
- If a deploy runs into problems during the run, what’s the best way for DevOps to be alerted? How can we address problems if they don’t appear until after the deploy is complete?
- What if DevOps assistance would make the deploy process smoother? Who do you pull in, and how?
As a result, we developed these PagerDuty utilities to be integrated into our deployment system. In this blog post, we will be covering an example set of uses that can help address these questions.
Getting help from a continuous deployment page
By connecting the
pgutils trigger command to a button on the deployment website, developers can click and instantly page the first person on-call. If something goes wrong, DevOps can be alerted to the exact deploy that needs assistance; or, if a deployer needs DevOps to be on-hand for the deploy, the on-call person can be quickly alerted to that.
For a less intrusive system, we can simply return a list of everyone on-call, and let the developers work out who the appropriate contact will be (using the
pgutils get_on_call command):
Assigning accountability to the deployer
Another problem with having developer-run deployments is assigning accountability to those developers, so they will be aware if their deploy manages to go awry. To this end, we can use the
pgutils set_override command.
By passing options into the script, we can easily assign someone to the on-call schedule based on project and user’s email (which are already recorded):
These are just a few examples of measures you can take to both limit the time DevOps is involved in the deploy process, while also making sure that developers are able to get all of their deploys out in a timely way. For more information, you can visit the Github repo at: https://github.com/Tapjoy/pagerduty_utils
Think this is interesting? We're hiring! See our current openings here.