Learn about the technology, culture, and processes we use to build Hootsuite.

Everyone — our software developers, co-op students, high school students — 'works out loud' about the tools we use, the experiments we ran, and lessons we learned. We hope our stories and lessons help you.

Recent Posts:

As Hootsuite’s products have grown, we have started to move away from a single monolithic project to a suite of microservices. Each of these microservices has an API that we use to hook all these disparate systems together. Like any large software system, we need to test these services end-to-end in order to ensure that they function correctly as a whole and to quickly find issues when one service changes in a way that affects its integration with others. To that end, we created Wasabi.

What is Wasabi?

Wasabi is a NodeJS app designed for end-to-end API testing. It uses Mocha to run tests and Chai as an assertion framework. We also designed a custom HTTP client module, as well as a custom Consul module that we use to lock the fixtures used in our tests.

Why did we create Wasabi?

Hootsuite had a number of previous methods to test our APIs, and none of them really did what we wanted. One of our first attempts was a combination of PyUnit tests attached to an older custom HTTP client that never worked very well. It had big problems trying to run tests concurrently and cleaning up our test setup after a failed test, and pretty quickly we decided we needed something better.

Another solution that we used for awhile was Runscope, a third-party product that allows users to create tests through a GUI. Runscope worked fairly well, but because it was a third-party product, we weren’t able to integrate it with our source control and we had to rely on external support more than we liked (plus it came with a monthly fee).

We also have a number of Scala integration tests, many of which are still in use. These were our best attempt at writing good tests before Wasabi, but parsing JSON is a lot more cumbersome in Scala than in Javascript. Eventually, we decided we needed something better, so we created Wasabi.

Wasabi was designed to solve all the problems that existed in our previous tests. It takes care of the concurrency and cleanup problems that plagued our Python solution, and developing it in-house means we don’t have to rely on an external product that’s hard to integrate with our code. It’s much easier to parse JSON responses with Wasabi then it is with Scala, which makes our tests easier to write and understand. Wasabi can perform end-to-end testing for all of our external APIs more reliably than previous solutions, and it has a fun emoji to use on Slack.

How to write end-to-end API tests?

Regardless of whether you use Wasabi or some other API testing framework, the steps required to write API tests are fairly consistent. There are two major elements to API testing: creating fixtures and writing the tests themselves.

Fixtures:

Fixtures refer to the data used in the API tests. At Hootsuite, that usually means Hootsuite accounts with access to specific social networks or organizations, but it could be whatever type of data is controlled by a given API. The important thing about fixtures is that they are fixed. Your tests will be verifying that the fixtures contain the data that you expect, so it’s important that they are not changed outside of tests, or you risk your tests failing due to outside interference. At Hootsuite, we have designated test accounts on each of our production environments that are used for specific test suites only and aren’t touched otherwise. We also lock each test account using Consul before running tests that hit them, so we can be certain to avoid outside interference while our tests are running.

Writing the tests:

After you have your fixtures, you can move on to writing the tests themselves. A happy path Hootsuite API test can generally be broken up into the following components:

  • Locking and verifying the account used for the test.
  • Authenticating the test account.
  • Echoing the API (i.e. hitting an endpoint that does nothing except respond with the same message you sent it. Useful for troubleshooting network problems when your tests aren’t passing).
  • Setting up external dependencies, such as bins for webhooks.
  • Hitting your endpoint.
  • Verifying that the response has the shape and content that you expect.
Below is an example Wasabi test that hits all of those steps. If you’re not using Wasabi, some of the specifics might be a little different, but the general structure will be very similar.

Example Wasabi test

The meat of the test is the last step, verifying the response. This step should be overly detailed, so you will know exactly what’s wrong when your test fails. Good practice is to start with verifying that the response looks as you expect at a high level, and then work your way down. A typical verification hierarchy would look like this:

  • HTTP response code is correct
  • response has a body
  • the body contains a data array
  • the data array is not empty
  • the data array has the fields you expect
  • each field has the content you expect
What are the current issues with Wasabi?

Wasabi runs end-to-end tests over Hootsuite’s internal network. That means it is heavily reliant on the stability of that network to run correctly. Slow connections can cause timeouts (this is especially a problem on our terrible wifi). Upgrades to seemingly unrelated projects can cause tests to break unexpectedly, which is especially a problem in our constantly-changing dev environment. Often, it’s difficult to tell if a new test failure is due to changes in the code or due to unrelated changes in our infrastructure. These aren’t necessarily problems with Wasabi itself, but they do place restrictions on how we can use it. We don’t run Wasabi tests on our dev environment at all, instead using it as a final check on production before we push an update to all users. This has helped lower the time we spend fighting with failing tests, while still ensuring that our product is tested thoroughly before release.

About the Author

Jacob Brunette is a Software Developer Co-op with the Platform team at Hootsuite. Jacob studies computer science at the University of British Columbia. Connect with him on LinkedIn.

Take a look at this piece of code:

1
2
10 PRINT "Hola, amigo!"
20 END

That was the very first two lines of code that I ever wrote! I was 7 years old, and when I ran this program my life completely changed!

This is why I love Hackathons! It brings back the exact same feeling of excitement that, as a 7-year-old, I got when I realized that I could make a computer do pretty much whatever I asked it to do! That’s why you should “hack” too! It’s all about seeing things that we do on a daily basis (like coding) but imagining that we are doing it for the first time so that 7-year-old passion always comes forward and shine on everything you do.

This week and next is our 2017 Hootsuite Hackathon, and for us, that’s the perfect opportunity to unleash that curiosity, determination, and passion! and be a 7-year-old again! I’ve been lucky enough to participate in the last two company Hackathons, and they have brought some of the best experiences in my career. They will always be my “favorite time of the year!

I really recommend participating in Hackathons whenever you can. It’s lots of fun! No idea is silly when it comes to hackathons, that’s the fundamental principle of brainstorming and how true innovation begins.

Here are a few tips that I learned from previous hackathons and that I wanted to share for anyone thinking about joining one:

  • If you are having a hard time coming up with ideas, pair up with our Product and Design team members; they invest a lot of their time brainstorming on how the future of our products may look like, so they could have some cool stuff in their “ideas drawer” for you!
  • Try to work with people from other teams. This is a great opportunity to get to know talented people that don’t necessarily interact with you on a daily basis. #newfriends
  • Hackathons are not just for Engineers. Last year I worked with designers and the year before we brought up Marketing. It is amazing what fresh eyes can bring to your project. We are always surrounded by so much talent outside of engineering and a lot of that talent is more than willing to participate.
  • Have tons of fun!
Hack away!

 

About the Author

Rei Colina is a Lead Software Developer. He leads the Labs Team at Hootsuite. Get in touch via Twitter @rei_colina

After any outage or severe incident at Hootsuite, we take a closer look at the event to uncover valuable lessons and data. We do this by working backwards, asking “Why?” to find the most likely root causes for the problem. Then we put in place incremental improvements to bulletproof ourselves against a reoccurrence.

Before I explain our mindset, practice, and step-by-steps, let me tell you a story.

Oh Shit

I’m sure you’ve felt this way before…

On April 6th, 2017, my spidey-sense started to tingle, my hands began to sweat, and I had that sinking feeling in the pit of my stomach: all the physical and emotional signs of having done something wrong.

Turns out, I brought down part of our site for 39 minutes. More specifically, our users were unable to post to LinkedIn or to add new LinkedIn accounts. I had inadvertently deleted the Hootsuite app from the LinkedIn Developer Portal because I thought I was deleting my developer access to the app. Fortunately, for our customers, this happened in the evening around 6:30PM PDT when our site is not under heavy load.

What happened next says everything about our culture.

Now what?

Everyone came together.

  • One of our Product Managers, helped me verify the degraded functionality.
  • Our VP Engineering calmly directed me to our Strategic Partnership team who have a relationship with LI.
  • Our SVP Strategy and Corporate Development tried to get in touch with a contact at LinkedIn while I submitted a support ticket to LinkedIn Developer Support.
  • One of our advocates from Customer Support let us know about the first customer reports of a problem in our Slack #outage channel.
  • Developers, Operations and others from across the company immediately responded in #outage and offered to help

The Fix

Turns out emailing LinkedIn Developer Support was the key to a fast resolution. Danthanh Tran from LI was able to “undelete” our Production, Staging, and Development apps in a matter of minutes.

Whose fault was it?

That’s a trick question. As my cold sweats told me, I felt responsible, but no one is to blame. This is key to the culture of a team that improves systems quickly.

The next day, a member of our Ops team facilitated a blameless post mortem called a “5 Whys” that focused on the timeline, data about the event, what went wrong, and how we can fix our system so the same issue doesn’t happen again.

Note, the words fix our system, not how we can fix me. Like most humans, I’m well intentioned but fallible. Many of us could have made that error. In fact, having personal social network accounts able to administer the one key connection to our social networks has been a time-bomb waiting to happen. This event is a catalyst for long-overdue change.

“Human error is a symptom—never the cause—of trouble deeper within the system” – Dave Zwieback in an article on First Round Review.

So, the question is how can we bulletproof our system so the next person doesn’t have the same inadvertent nightmare? A more humorous way of saying this is from Dan Milstein, in How to Run a 5Why’s (with Humans not Robots): “Let’s plan for a future where we are all as stupid as we are today” 🙂 Focusing on the system instead of the person is a way to remove the stigma of speaking up when we feel guilt for an accident. This key because accidents contain valuable information and should be “seen as a source of data, not something embarrassing to shy away from” via John Allspaw.

Next, we post a summary of our 5 Whys, on our internal tool called Morgue (also from Etsy) and take actions to abate the probability or a repeat event.

A Note on the title of this outage.

Two people pointed out the incongruity of saying we’re blameless but using a name in the title “That time Noel Deleted LinkedIn”. Let me explain. This is the only time we put a name to a 5 Whys. It was done as a satirical tribute to me, expressly for my contributions to our culture of blameless post-mortems.

Our Mindset and Culture on 5 Whys

To surface all the valuable data our culture must remain “open, with a willingness to share information about … problems without the fear of being nailed for them.” – Sidney Dekker in The Field Guide to Understanding ‘Human Error’.

Trust by default

  • Believe that people are doing the best they can with the information they have.

Provide psychological safety

  • How many of you would have admitted to causing a problem that brought down production? Do you feel you can take the risk to tell people without feeling insecure or embarrassed? Censoring ourselves means we limit what we can do.
  • As my colleague Geordie put it: “only people who feel psychologically safe in a team and can be themselves, are capable of doing their best work.”

Helpfulness is the secret weapon

  • Helpfulness feels great and it is an indicator of highly productive teams. Margaret Heffernan in her TED talk speaks about an MIT study of teams, where teams were given very hard problems to solve. The really successful teams were not the teams with the highest aggregate IQs. They were, in fact, the teams that were the most helpful and the most diverse: had the most empathy towards one another, the one’s without a dominant voice, and the ones with the most women.

Continuous Improvement supported by data

  • One of our values is Build a better way. To do this we have to make time for reflection then follow it up with adaptation. Supporting data is key. Tracking metrics and severity in a single place means we can use it to add context, to make decisions, and to measure our improvement.

How we got here

We arrived at these practices because people at other companies ‘worked out loud’ and showed us. Then, we iterated towards our practices based on our context.

The following list is from invaluable thinkers who influenced our mindset, practices, tools

Step-by-step

  1. Something happens… an outage or incident

  2. Resolve it.

  3. Read the articles above. If you read just one, read this one.

  4. Set a time for a post-mortem. The sooner the better. Grab a whiteboard.

  5. A note about where to hold it. A whiteboard is in public place, not in a meeting room. We also leave the details up until the next 5 Whys. This is a visible reminder of openness and it invites participation from passers-by who provide input based on their past experiences.

  6. Invite the people or persons who… (taken from Bethany Macri‘s talk on post-mortems):

    • introduced the problem
    • identified the problem
    • responded to the problem
    • debugged the problem
    • anyone interested

  7. Conduct a 5 Why’s (blameless post-mortem). Focus on the system not the person.

  8. Like Dan Milstein says, start with humour to get participants to open up. Relative to other disasters… How bad is this? Did the site go down? Accidentally post from our main twitter handle? Lose customer data? Send an email to all our users with the greeting “Hello FName!”

  9. Collect metrics

    • Timeline
    • MTTR: Mean Time to Resolution
    • MTTD: Mean Time to Discovery

  10. Drill down
    1. Ask “Why?” at least five times.
    2. Each statement should be both verifiable (how do you know?) and not compound (single cause). An example from Five Whys – How To Do It Better:
    • Example problem: SUV Model Z exhaust system rattle
    • Why? Change of position of bracket results in vibration <– COMPOUND
    • Why? Exhaust pipe vibration <– SINGLE
  11. You must be able to work up the chain (last ‘why’ to the first ‘why’) to verify the causation.

  12. List the improvements you can make to bulletproof the system

  13. Ask people to commit to completing an improvement.

  14. How do we arrive at action items and commitments? Organically. As a heuristic, point to each “why”, and to each step in the timeline, then ask how we can improve the process, technology, and training for each item; and who will take it on.

  15. Publish it in Morgue or in the most visible communication channel.

  16. Follow-up on that channel as you make the fixes and share your learnings.

    Too soon? 🙂

    Thank you

    To everyone who helped with this outage. To Jonas, Beier, Geordie, Tyler, James, and Trenton for their contributions to this process and their help putting this article together.

    About the Author

    Noel Pullen 200x200 Noel focuses on culture, employee engagement, technical community involvement, and training for Hootsuite’s technical groups. He loves to exchange ideas and would like to hear how you do these things at your organization. Get in touch via LinkedIn.

Henry Ford once said, “Coming together is a beginning, staying together is progress, and working together is success.” It is based upon this philosophy of embracing collaboration from start-to-finish that my team, as well as numerous others at Hootsuite, have adapted an additional role in our Agile methodology.

To provide some background for those unfamiliar with Agile, an Epic is a large unit of work which, when completed, will provide a lot of value to the customer. Epics are then broken down into stories and tickets/tasks which developers will commit to and complete.

Every developer is encouraged to work on whichever task is highest in priority allowing work to be fluid and ensure each developer is well-rounded. However, each sprint, there can be numerous Epics being worked on, as well as numerous more being planned in the backlog, which often makes it difficult for product owners to maintain an accurate idea of the current progress of each based on small, fragmented updates from each developer at scrum. Further, the process of conceiving a new feature often gets muddled as it is passed around between design and growth and management before finally arriving at the engineers. The solution to all these problems and more? The Epic Champion. Read More …

I was honored to work on the Test Tooling Team at Hootsuite since January, and one of the main projects that I actively participated in is the Arbiter Tests Management System, a project that provides a user-friendly way for QAs and Developers to manage the tests to be run on different build pipelines in real time. By allowing the users to directly toggle tests from the UI, Arbiter has significantly reduced the resolution time to tests failures in the organization.

As the program aims to optimize developer’s experience, we focused not only on the practical convenience of the program, but also on the design of the user interface that could create a pleasant visual effect for the developers. With this in mind, we have designed 5 different themes ranging all over the spectrum: dark, light, oreo, tiramisu, and azure.

While React has been the de-facto standard for front-end development at Hootsuite, we have also chosen React as our front-end language to effectively build the visual effects we intended to achieve.

CSS to LESS

When Arbiter was first built, we stored our styles into old-fashioned CSS Stylesheets, where themes are loaded as a stylesheet link element in HTML, and switching stylesheet involves simply changing the link of the style element with a Javascript function.

In February, I completed the process of refactoring CSS Stylesheets into LESS Stylesheets to match the organization standards at Hootsuite. This transition results in a much more efficient integration with React. The stylesheets are now loaded directly into the components they style with less-loader so that we don’t have to work with stylesheets in raw CSS.

BUT, alas, every rose comes with its thorn! Now that stylesheet loading is handled by less-loader, we can no longer explicitly manipulate the stylesheet links to switch themes 🙁

After two days of desperate research, I experienced a lot of frustration with the lack of solutions on this topic online. Through this blog post, I would like to demonstrate this simple process that I came up with so that people in my shoe could save their research time. It is also a testament that sometimes it is not necessary to rely on more advanced packages to solve issues, simple Javascript with some thoughts to it just does the job equally well!

Component Structure

To make things simple, I have separated all the theme-loading related functions into its own React component named ReactThemeLoader. The overall structure of this component involves initially loading all the available themes, disabling all the themes, and re-enabling the current theme.

The mechanism of this system relies the fact that stylesheets loaded onto DOM can be toggled on and off by setting the boolean property “disabled”.This property gives us the flexibility to load multiple themes onto the page at once by enabling the current theme and disabling others, without having them interfering one another.

In componentWillMount, all the themes are loaded and disabled initially. The index of the theme stylesheets are tracked here and recorded as key-value pairs into a dictionary called themes in state.

After the components are mounted, in componentDidMount, the program reads the theme name stored in the cookie and and sets the theme by calling setTheme(). In case the cookie is not present, the theme will be set to default theme (in this case dark), which is handled by the setTheme() method.

In the setTheme method, the stylesheet indexes of the old theme and new theme are simply retrieved from the dictionary themes and toggled by setting the “disabled” property of the stylesheets as shown above.

The UML Diagram for the theme-loading process is shown below:

NPM Package

To make this React Component more reusable, I have published an NPM package: react-theme-loader (http://ow.ly/U9Lt30asuTX) as a ready-to-use open-source React package for you. Below is where the tutorial actually starts

Install the NPM Package

Under your project directory, run:
npm install react-theme-loader –save
 

This command will install this node package to your project and add it to the dependencies in package.json

Import the React Component

In your top-level React components where you want the theme styles to take effect, import the package and render it as a component

When rendering the ThemeLoader, there are certain properties we need to attach to the element; the above is just an example, please read the instructions below on filling in the props and do NOT copy the above exactly.

Required
“ref”: pass in “themeLoader” exactly so that you have access to the functions provided by the component.

“supportedThemes”: pass in an array of strings that represent the names of the theme files (without “.less”).

Example:

“themeDirectory”: pass in the relative path of the directory of the theme files in “supportedThemes”

Optional
“fonts”: pass in the relative path of the fonts file. This file should contain ALL fonts used in the themes, and do NOT import any fonts in the theme files since they mess up the indexes of loaded stylesheets. If “fonts” is not specified, no fonts file will be loaded.

“defaultTheme”: pass in the name(string) of the default theme, which will be loaded initially when no theme cookie is stored. The first theme in “supportedThemes” will be the default if not specified.

“themeCookie”: pass in the name(string) of the cookie where you wish to store the current theme name, so that the browser will remember the selected theme. “CURRENT_THEME” will be used if not specified.

Using react-theme-loader

Switching Themes
In the React component where ThemeLoader is rendered, simply use

to switch themes, where theme is the name(string) of the theme you wish to switch to.

When a theme is loaded, the ThemeLoader will render the following to HTML:

Cookie

Currently the theme information in our system is completely relying on un-encrypted browser cookie, named “CURRENT_THEME” by default. The package gives you an option to pass in the cookie name you wish to have.

Although theme information is not sensitive, it still brings inconvenience and leaves holes for security issues. We are currently looking into other ways of improving this features.

React-Theme-Loader 2.0 !!!

Currently, the react-theme-loader package is still in its rudimentary stage and a lot of customized features can be added to the component. The ones listed below are in progress, please feel free to leave any ideas below in the comments

 

  • Customized way to store current theme information
As mentioned above, the current way of storing theme information in cookie isn’t ideal. In a lot of applications, it is more ideal to store those to a user account. Instead of doing the cookie handling in the component itself, the component will be able to read in a customized function by the user to handle the job.

 

  • More robust import handling
Currently, you would have to store all the imported fonts into a separate fonts file for the component to read the correct indexes of other stylesheets, since importing fonts in a stylesheet causes index problems. In the future, more investigation will be made to ensure that adding “import” statements in .less files will not cause the program to break.

Conclusion

Completing the transition from CSS Stylesheets to LESS Stylesheets is one of the first refactoring process I have worked on during my time at Hootsuite. While I have been through a lot of frustrations and desperation throughout this process, I was forced into discovering more hidden features in the world of programming (leading me to make my first npm pacakge!) Looking back at the process, I would consider it is a tremendously helpful experience in the long run with major improvements made to the system.

About the Author

Sonny Yu was a co-op student on the Test Tooling team for the spring term of 2017. He studies Computer Science and Business Administration at University of Waterloo. Connect with him on LinkedIn!

Loading ...