By Mark Eijsermans on August 25, 2015
Continuous Integration (CI) and Delivery (CD) are a core processes within our engineering team. CD is hard to get right, and we pride ourselves on following best practices. In the cultural sense, we imbue the central values and DevOps culture in general: continuous improvement, tight feedback loops, automation, education, blamelessness, and autonomy.
In fact, it can be argued that CI is an absolute baseline requirement in modern software development. It is the hub for all spokes that make up engineering (devs, ops, QA, management). The build and release pipelines are therefore a mission critical production distributed system, and should be treated as such. But what happens when the pipeline (or parts of it) goes down? Important fixes can’t be pushed out, developers get blocked, visibility is lost, and everyone notices. Worse, when the problem is resolved, the commits have piled up and an important tenet is broken: small incremental changes vs. large deployments.
When starting out as a monolithic app developed with a small team, you can easily get away with just configuration management and a simple build server (eg. Ansible & Jenkins). But as time moves on, the product, team and company grow, and that monolith is half way transitioned to a handful micro-services.
Last ½ year we took a step back to look at things with a bit of perspective. We can’t just hire someone to monkey around with Jenkins and expect our growth problems to go away. A deeper re-examination and contemplation of the current issues (and future requirements) needs to take place. With a solid design philosophy, implementation more easily falls into place and decisions are made within the proper context. So, in the spirit of openness, we’re opening up our dev process in the hopes that we get insights from unexpected places and that we unexpectedly help others by providing lessons learned.