Abstracting Marathon Deployment Details from Microservices
Moving to Docker and Container Orchestration
At Hootsuite, we are moving from in-place deployments with statically provisioned hosts to running Docker containers on an orchestration platform. This platform manages the running containers and handles scaling and failure recovery. Transitioning to Docker will allow us to build microservices more quickly and with reduced operational overhead. We’ve chosen Marathon and Mesos as our orchestration platform (other options were Docker Swarm, Kubernetes, ECS, and Nomad).
How do we deploy with Marathon and Mesos?
To deploy a service to Marathon/Mesos we need to HTTP POST a JSON configuration file (marathon-config.json) that specifies attributes like the Docker image to run, the number of instances to deploy, what compute resources should be allocated, how to health check the container, etc. When we first started out, we kept each service’s configuration file in their respective git repositories. During deployments, Jenkins would clone the repo and POST the JSON directly to Marathon.
Here is an example of the JSON config:
However, there were a number of issues with this approach:
- Marathon-specific configuration details were embedded in each service’s repository. Service developers had to ensure their JSON configuration had specific field names and structure necessitated by Marathon, leading to more mistakes while writing it. When completed, the JSON was lengthy and difficult to read.
- As platform developers responsible for working on the Marathon platform, when we wanted to make changes to Marathon-specific configuration details, for example adding a new option
to1-sto specify how health checks were run, we had to make pull requests to every single service’s repo.1curl
- In general, services were deeply coupled with Marathon and it would have been difficult to make larger changes in the future.
In order to address these issues and support a growing number of services, we decided a middle layer between service-specific config and Marathon was necessary: an HTTP application that would take service-specific config and hydrate it with Marathon-specific config before passing it on to Marathon. This middle layer would allow developers to keep config like the number of instances and memory allocation in their repository config files while abstracting out the config they didn’t need. We called this middle layer Skytrain.
Let’s look at the example of health checks again to illustrate how this is useful. Previously, each service had this JSON config in their repositories:
This config held a lot of Marathon-specific data like using curl for the health checks and which options should be passed to curl. When we realized we had to add a new option to curl, we had to submit a pull request to every single service’s github repos to make that change.
Instead, with Skytrain, the services have this YAML config (config.yml) in their repositories:
Skytrain will then take this configuration and add in the Marathon-specific details like curl options. Now, when we want to change the options to curl, we just have to do it in Skytrain.
Skytrain reduces deployment complexity by enabling services to only keep track of how their service is different than others — services are abstracted from needing to worry about the details of deploying to Marathon.
Skytrain Implementation OverviewSkytrain consists of three components:
- YAML config files: Developers only need to specify deployment details specific to their service. These YAML config files require setting team names, number of instances to deploy, health checks, docker image, compute resources, etc.
- Skytrain Scala Service: The Scala service accepts and validates deploy requests containing service-specific config and transforms them into the complex JSON required by Marathon. It figures out which environment to deploy the service to based on the environment specified by the initial call to Skytrain in the YAML config.
- Python CLI: Rather than directly POST’ing the service-specific YAML file to Skytrain, we added a Python CLI that combines the team-specific YAML with a generic YAML file containing deployment details common to all services. The python cli then makes an HTTP request to the Skytrain scala service.