Managing servers at scale has always been a difficult task. As your environment grows, it becomes much harder to maintain consistency, while maintaining flexibility. AWS provides a much needed starting point with their autoscaling feature (http://aws.amazon.com/autoscaling/); however, over time this too can grow unruly.
While the CLI that Amazon offers (http://aws.amazon.com/cli) had proven useful, we wanted some features focused on configuration management best practices. To this end, the Tapjoy Operations team has developed TASS - the Tapjoy AutoScaling Suite.
One of the first problems faced by any infrastructure at scale is how to manage change. Virtualized infrastructure makes this a bit easier by making everything software; unfortunately, when you're running command line tools with flags for configuration parameters, you are still left with the problem of managing the change.
To illustrate this, we need to first walk through what it looks like (borrowing from Amazon's examples):
You could write a script for each autoscaling group and launch configuration pair. This approach couldn't scale with our growth, so we developed a new way:
The real magic behind this is that we've separated the code from the data, placing our configuration in a separate store. To replicate the behavior above, your configuration file may look something like this:
This is a very simplified configuration, and your environment will likely need more config options. All of them are documented in TASS CONFIG_OPTIONS file.
Consistent Configuration and Setting Sane Defaults
TASS uses a hierarchy of yaml files to configure a given cluster: defaults, <environment> , and <cluster> (lowest-priority to highest). If you fill the defaults.yaml and <environment>.yaml with as much configuration as possible, then the cluster-specific yaml file will be only require override values and could be just a few lines long.
This additionally extends to the userdata scripts, which are ERB formatted.
To see an example of what the yaml files look like, you can reference the fixtures in the TASS repo.
While TASS offers consistent bootstrap scripts, it is also highly flexible. You can write your bootstrap to fit your needs and fill in variables by adding them to your cluster configuration. An example use case would be to specify a chef server in either your environment or defaults config file.
TASS is a full-featured command-line application that we've only begun to dig into here. For more information, please visit the GitHub project at: http://github.com/Tapjoy/tass.
Think this is interesting? We're hiring! See our current openings here.