Day 1: Push to Production
We have a norm for onboarding new engineers on Day 1: Push to Production. This norm is a powerful experience for our new people and a litmus test for our systems.
The Experience and Why it’s ImportantPushing to Production on Day 1 is a small and early exposure to an engineering culture that values learning, trust, collaboration, and a bias towards thoughtful action.
Always Be Shipping. As software engineers, this our role, this is why we’re here, this is how we make an impact and help others – like our customers.
This is a hands-on experience. There is an important difference between passive learning and active practice. Reading a wiki page or diagram about deployment steps and hearing our philosophy of “anyone can deploy to Production at any time, from anywhere in less than five minutes” develops knowledge. The act of deploying helps someone understand how to change our product and how it feels to make that change.
This shows the speed at which we operate. Continuous Delivery – shipping to Production multiple times per day – means we can get something useful into the hands of our customers very quickly. We pushed 2015 times in 2014, an average of 8 times per day.
This says ‘we trust you’. Our engineers are responsible for delivering code the last mile and accountable to how it affects our customers. We try to convey the magnitude and gravity of shipping to Production, but at the same time, remove the fear of deployment. This shows the value of small changes. The smaller the change the easier it is to debug the problem. There is less risk in deploying numerous small changes than bundling those changes together into a bigger deployment.
This is an accomplishment. Some of our engineers come from places where it takes a month or six to deploy, some where no deployments happened at all during their co-op terms. This is something to point to at the end of the day.
This is a reason to celebrate. Lately, as soon as a new person on our Platform team deploys to Production on Day 1, the team breaks into applause (then everyone else checks the “Why we just clapped” room in HipChat).
This is a constraint. The Day 1 boundary forces us to work backwards and examine everything that is necessary to make a deploy. Just keep the critical steps, and eliminate unnecessary tasks or delay them to a future date, like filling in tax forms or office tours.
This is a Litmus TestWhat kind of code modification is made on their first day? Add your name to a humans.txt file; or add your name to a config file, and calling our /about/humans endpoint to see your ‘name in lights’. This trivial change is a simplification of a complicated procedure that proves out our cultural practices, our technical systems, and acts as litmus test for our purpose: delivery of code to Production – either you push to Production or you don’t (and if not, we examine the reasons why).
Cultural practices include:
- ‘rubber ducking‘ your intent with your training guide (understand the problem, and talk through your “flight plan” prior to coding)
- adding a feature flag (we call this “dark launching”)
- using JIRA
- asking for a code review
- two formal Pull Requests with peer code-review and signoff
- deploying-followed-by-system-health-monitoring to Development, Staging, and then Production, respectively.
- seeing your name on humans.txt or returned by our /about/humans end point
- laptop setup: reviewing our OS X images so each Engineer has a laptop with tools necessary to checkout, modify, and deploy code.
- accounts setup for tools like GitHub Enterprise, monitoring sites, and wikis
- access to necessary services and servers to be able to deploy
- requirement that our processes meets SOC2 Type 2 security requirements
What happens if someone breaks the norm?We learn from it. A ‘hard and fast rule‘ like at Etsy is our aim, but for a while in 2015 the practice of shipping to Production slipped to Day 2. That natural drift can happen unless the reasons for shipping on Day 1 are:
(a) explained clearly and publicly – such as in this post;
(b) repeated enough to gain traction (in other words, the ‘why’ is repeated by more than a single voice). The value of shipping on Day 1 is communicated during a training session to Training Guides on the Wednesday prior to a new person starting;
(c) there is a mechanism in place to learn from our misses: an individual and/or a group responsible for collecting feedback from new hires and actioning it. For us, this happens through our Onboarding Guild.
What do non-Engineering groups do?
Ideally, it’d be a great experience for non-engineering groups to be able to push to Production too but that’s not always the best experience or test of their systems. Our groups – like IT, Operations, Security, and those working on our native mobile apps – have the challenge of coming up with something significant and memorable that others can celebrate. For example, our IT team tailored an experience to a common situation handled by all IT staff: on Day 1 new people are given a un-initialized laptop and work towards setting it up such that they can post a picture of themselves – with their newly setup laptop – on our internal social network.
What are your norms?Why do they exist and how do they help you, your people and your organization? What do you do to teach new people about your culture, practices, and to ensure your systems work as expected (and if not improve them)?
To Etsy for the inspiration. To all the peeps here that make this practice and these systems possible. To Lindsay, Mike W, Mike H, Kimli and for their editing.
About the Author
Noel focuses on culture, employee engagement, technical community involvement, and training for Hootsuite’s technical groups. He loves to exchange ideas and would like to hear how you do these things at your organization. Get in touch via LinkedIn.