Posts from July 2015

“You must feel the Force around you; here, between you, me, the tree, the rock, everywhere, yes.” – Master Yoda

The Danger of Team Debt

At her PyCon 2015 keynote, Kate Heddleston explains how a familiar engineering concept – technical debt – applies to any growing organization. Any technical system can accrue technical debt as a consequence of bad design. Heddleston argues that organizations can also accrue ‘team debt‘ as a consequence of bad design: where each person added to your team eventually decreases overall team productivity. Productivity drops because each new addition lacks an understanding about the team’s processes, cultural norms, how to do their job, corporate values, code standards, architecture, and more.

Steve Blank sums it up as “all the people/culture compromises made to ‘just get it done’ in the early stages of a startup.” It’s like death by a thousand cuts – each person’s inefficiencies compound to a point where their time and effort spent navigating your ‘system’ outweighs their time and effort spent shipping code.

This was exactly our situation in our Engineering group two years ago. Our team had tripled in size from 13 to 39 in the span of two years and was slated to double again to 78 in 2014. So much of our ‘just get it done’ approach lead to misaligned expectations and lack of understanding of our code base, our practices, and our culture. Symptoms of the problem trickled in to me periodically, but the depth of the situation really hit home when someone I had hired resigned and cited some of these issues in their exit interview.

That event radically shifted the way I looked at introducing new engineers.

Lechon Kirb photo via

Read More …

Why Bother?

The obvious question that initially emerges is WHY? Why bother with all the hassle of migrating the code away from GitHub and maintaining the instances that provide the service on our own?

First of all, GitHub Enterprise offers us a more secure way to store the sensitive parts of our codebase by bringing the repositories inside our VPC. Furthermore, because the instances hosting the code and the ones using it are now much closer together, code provisioning can be done much faster.

Secondly, it is better to have some control over your downtime than to be at the mercy of GitHub (or any other service for that matter), which can lead to canceled deployments or angry developers who cannot pull their code. As frustrating as downtime is, it’s not a matter of ‘if’ it will happen, but rather ‘when’ will it will happen.

We chose GitHub Enterprise over other forms of repository hosting providers, like GitLab, because of the reasons above and because our developers were already familiar with the interface, features and embraced the GitHub flow. GitHub Enterprise is easy to update and it had a better API.


Will it stand up in our Production Environment?

Before we could actually start using GitHub Enterprise in production, we needed to see if it could support our blue/green deployment system (more details on this can be read here). This meant that it should stand hundreds of instances that needed to pull their code from the repository, simultaneously.

To test this, we have used two r3.xlarge memory optimized instances, offering 32GB of memory and 4 vCPU, in a replication setup. This is what GitHub recommends for a seat range of 500 to 3000 people.

Read More …

The art of storytelling has existed since the dawn of time, and is one of the few things that has both changed drastically over the years and yet remains essentially the same. Technology has given us the printed word, theatre, movies and television, and more – but it can all be traced back to people gathering together to share experiences.

@mixhellereid shares her story - photo by @ivancouverite
@mixhellereid shares her story – photo by @ivancouverite

Hootsuite Labs is working to bring Vancouver’s tech community together to share stories about life in technology. Labs is the force behind some of our new products and initiatives such as the new Hootsuite Suggestions app and Hootlet, the Chrome extension that allows you to share content from anywhere you browse, and they’ve been very busy lately: in addition to their new blog, they’ve launched Venture This!: True Tales from the Tech Frontier with Rain City Chronicles. In their own words:

Venture This! is a live event series and podcast featuring real stories about life, love, and work experienced through tech-coloured glasses. Stories are told in front of a live audience, are 100% true, and offer a range of perspectives, from tech tycoons and young upstarts to the adventures and misadventures of total tech newbies. Every night reveals new insights into the strange and wonderful mysteries binding our connected world.
On Monday, June 29th, Venture This! had its inaugural event at Hootsuite HQ1. The topic was “Rejected!”, and featured storytellers from Vancouver’s large and diverse tech community sharing their stories of rejection in the tech world. These stories will soon be available as a podcast for you to enjoy, but in the meantime, check out the Tech Vibes review of the event:
The above is only a whisper of the richness of the personal accounts at the Venture This! event. With humour and honesty, the presentations provided relatable, cathartic, and cautionary stories of adversity, of external and internal origins, and coming out the other side alive.
Don’t miss out on the next Venture This! event! Sign up to be notified when the next session is announced, and come hear about the personal triumphs – and spectacular failures – from some of Vancouver’s best and brightest in tech.

Storytelling at Hootsuite! Photo by @mtippett
Storytelling at Hootsuite! Photo by @mtippett

Picture this: a group of new engineering Owls situate themselves pairwise at temporary desks arranged in our yoga room. With varying technical backgrounds, we ready ourselves for participation in two weeks of Scala & Akka immersion to acquire a certain level of fluency. After reading numerous books on the subject and completing Coursera’s Functional Programming Principles in Scala, what could a Scala & Akka Dev Factory offer us that we hadn’t already encountered?

As it turns out, the course offered more than just a great technical education. Our trainer brought to life the lecture notes, code samples, exercises, and group projects with both social learning practices as well as on-the-fly adjustment of the content in order to reinforce concepts in a really unique way. Some of these techniques included positive reinforcement from daily public code reviews, and pair programming with random partners who had varying skill levels and communication styles. Our daily input into the course also let us shape our classroom experience by driving the expansion of the course topics where we felt we needed it most. These shared learning experiences gave me and my cohort a kaleidoscopic insight into the course material.

June 15/2015 Cohort from left to right: Jonathan, Yasha, Philipp, Jason, Jodi, David, Scott, Mike S., Sim, Mike R., Ryan, Isha Photo Credit: Steve Song
June 15/2015 Cohort from left to right: Jonathan, Yasha, Philipp, Jason, Jodi, David, Scott, Mike S., Sim, Mike R., Ryan, Isha Photo Credit: Steve Song

Read More …

Upgrading a session storage system that services 5000 requests/second without downtime is no easy task. As part of continued efforts to harden our systems at Hootsuite, we moved from a legacy Memcached and MySQL session storage to a multiple availability zone Redis setup using Amazon ElastiCache. We ran into some issues along the way, but in the end we successfully migrated millions of sessions to the new system without any downtime. This post describes our strategy, the gotchas, and step-by-step of how we made this happen.


Read More …