Abstracting Marathon Deployment Details from Microservices

Moving to Docker and Container Orchestration

At Hootsuite, we are moving from in-place deployments with statically provisioned hosts to running Docker containers on an orchestration platform. This platform manages the running containers and handles scaling and failure recovery. Transitioning to Docker will allow us to build microservices more quickly and with reduced operational overhead. We’ve chosen Marathon and Mesos as our orchestration platform (other options were Docker Swarm, Kubernetes, ECS, and Nomad).

How do we deploy with Marathon and Mesos?

image01

To deploy a service to Marathon/Mesos we need to HTTP POST a JSON configuration file (marathon-config.json) that specifies attributes like the Docker image to run, the number of instances to deploy, what compute resources should be allocated, how to health check the container, etc. When we first started out, we kept each service’s configuration file in their respective git repositories. During deployments, Jenkins would clone the repo and POST the JSON directly to Marathon.

Here is an example of the JSON config:

However, there were a number of issues with this approach:

  1. Marathon-specific configuration details were embedded in each service’s repository. Service developers had to ensure their JSON configuration had specific field names and structure necessitated by Marathon, leading to more mistakes while writing it. When completed, the JSON was lengthy and difficult to read.
  2. As platform developers responsible for working on the Marathon platform, when we wanted to make changes to Marathon-specific configuration details, for example adding a new option
    1
    -s
    to
    1
    curl
    to specify how health checks were run, we had to make pull requests to every single service’s repo.
  3. In general, services were deeply coupled with Marathon and it would have been difficult to make larger changes in the future.

Enter Skytrain

In order to address these issues and support a growing number of services, we decided a middle layer between service-specific config and Marathon was necessary: an HTTP application that would take service-specific config and hydrate it with Marathon-specific config before passing it on to Marathon. This middle layer would allow developers to keep config like the number of instances and memory allocation in their repository config files while abstracting out the config they didn’t need. We called this middle layer Skytrain.

image02
Click to see full size

Let’s look at the example of health checks again to illustrate how this is useful. Previously, each service had this JSON config in their repositories:

This config held a lot of Marathon-specific data like using curl for the health checks and which options should be passed to curl. When we realized we had to add a new option to curl, we had to submit a pull request to every single service’s github repos to make that change.

Instead, with Skytrain, the services have this YAML config (config.yml) in their repositories:

Skytrain will then take this configuration and add in the Marathon-specific details like curl options. Now, when we want to change the options to curl, we just have to do it in Skytrain.

Skytrain reduces deployment complexity by enabling services to only keep track of how their service is different than others — services are abstracted from needing to worry about the details of deploying to Marathon.

Skytrain Implementation Overview

Skytrain consists of three components:
  1. YAML config files: Developers only need to specify deployment details specific to their service. These YAML config files require setting team names, number of instances to deploy, health checks, docker image, compute resources, etc.
  2. Skytrain Scala Service: The Scala service accepts and validates deploy requests containing service-specific config and transforms them into the complex JSON required by Marathon. It figures out which environment to deploy the service to based on the environment specified by the initial call to Skytrain in the YAML config.
  3. Python CLI: Rather than directly POST’ing the service-specific YAML file to Skytrain, we added a Python CLI that combines the team-specific YAML with a generic YAML file containing deployment details common to all services. The python cli then makes an HTTP request to the Skytrain scala service.

Impact

Skytrain enhances maintainability by allowing service developers to focus on what their service needs to run while abstracting away the underlying platform details. From the platform developer’s perspective we can now iterate more quickly and have more flexibility. For example, if in the future we decide to move away from Marathon to a different orchestration platform, changes need only be made within Skytrain while the services can remain untouched. Skytrain allows Hootsuite to create microservices faster while moving towards a true microservices architecture.
About the Author
image00Jerry is a Platform Developer co-op who is currently studying Computer Science at the University of British Columbia. In his spare time, Jerry likes going on hikes with his dog, playing ultimate frisbee, and cheering for the Vancouver Canucks.