API Testing at Hootsuite
As Hootsuite’s products have grown, we have started to move away from a single monolithic project to a suite of microservices. Each of these microservices has an API that we use to hook all these disparate systems together. Like any large software system, we need to test these services end-to-end in order to ensure that they function correctly as a whole and to quickly find issues when one service changes in a way that affects its integration with others. To that end, we created Wasabi.
What is Wasabi?
Wasabi is a NodeJS app designed for end-to-end API testing. It uses Mocha to run tests and Chai as an assertion framework. We also designed a custom HTTP client module, as well as a custom Consul module that we use to lock the fixtures used in our tests.
Why did we create Wasabi?
Hootsuite had a number of previous methods to test our APIs, and none of them really did what we wanted. One of our first attempts was a combination of PyUnit tests attached to an older custom HTTP client that never worked very well. It had big problems trying to run tests concurrently and cleaning up our test setup after a failed test, and pretty quickly we decided we needed something better.
Another solution that we used for awhile was Runscope, a third-party product that allows users to create tests through a GUI. Runscope worked fairly well, but because it was a third-party product, we weren’t able to integrate it with our source control and we had to rely on external support more than we liked (plus it came with a monthly fee).
Wasabi was designed to solve all the problems that existed in our previous tests. It takes care of the concurrency and cleanup problems that plagued our Python solution, and developing it in-house means we don’t have to rely on an external product that’s hard to integrate with our code. It’s much easier to parse JSON responses with Wasabi then it is with Scala, which makes our tests easier to write and understand. Wasabi can perform end-to-end testing for all of our external APIs more reliably than previous solutions, and it has a fun emoji to use on Slack.
How to write end-to-end API tests?
Regardless of whether you use Wasabi or some other API testing framework, the steps required to write API tests are fairly consistent. There are two major elements to API testing: creating fixtures and writing the tests themselves.
Fixtures refer to the data used in the API tests. At Hootsuite, that usually means Hootsuite accounts with access to specific social networks or organizations, but it could be whatever type of data is controlled by a given API. The important thing about fixtures is that they are fixed. Your tests will be verifying that the fixtures contain the data that you expect, so it’s important that they are not changed outside of tests, or you risk your tests failing due to outside interference. At Hootsuite, we have designated test accounts on each of our production environments that are used for specific test suites only and aren’t touched otherwise. We also lock each test account using Consul before running tests that hit them, so we can be certain to avoid outside interference while our tests are running.
Writing the tests:
After you have your fixtures, you can move on to writing the tests themselves. A happy path Hootsuite API test can generally be broken up into the following components:
- Locking and verifying the account used for the test.
- Authenticating the test account.
- Echoing the API (i.e. hitting an endpoint that does nothing except respond with the same message you sent it. Useful for troubleshooting network problems when your tests aren’t passing).
- Setting up external dependencies, such as bins for webhooks.
- Hitting your endpoint.
- Verifying that the response has the shape and content that you expect.
Example Wasabi test
The meat of the test is the last step, verifying the response. This step should be overly detailed, so you will know exactly what’s wrong when your test fails. Good practice is to start with verifying that the response looks as you expect at a high level, and then work your way down. A typical verification hierarchy would look like this:
- HTTP response code is correct
- response has a body
- the body contains a data array
- the data array is not empty
- the data array has the fields you expect
- each field has the content you expect
Wasabi runs end-to-end tests over Hootsuite’s internal network. That means it is heavily reliant on the stability of that network to run correctly. Slow connections can cause timeouts (this is especially a problem on our terrible wifi). Upgrades to seemingly unrelated projects can cause tests to break unexpectedly, which is especially a problem in our constantly-changing dev environment. Often, it’s difficult to tell if a new test failure is due to changes in the code or due to unrelated changes in our infrastructure. These aren’t necessarily problems with Wasabi itself, but they do place restrictions on how we can use it. We don’t run Wasabi tests on our dev environment at all, instead using it as a final check on production before we push an update to all users. This has helped lower the time we spend fighting with failing tests, while still ensuring that our product is tested thoroughly before release.
About the Author
Jacob Brunette is a Software Developer Co-op with the Platform team at Hootsuite. Jacob studies computer science at the University of British Columbia. Connect with him on LinkedIn.