Our Journey In Smoke Test Frameworks

Smoke testing is an important part in Hootsuite’s Build Pipeline. During my work term at Hootsuite, I got a chance to be a part of the change from a PhantomJS based framework to a Selenium one. The purpose of this post is to share my learnings from this journey — and that there is no so-called ‘best’ framework that will work in any situation. Choose the right tool for your business.

Overview of Web Smoke Tests

The smoke test is under the category of end-to-end tests which inspect the basic functionalities of a project. It is similar to the functional test but not as detailed.

In the practice with web applications, the standard way is to automate a browser (either a headless one like PhantomJS or a real one like Chrome/Firefox) and simulate the actions as if a user is actually viewing and clicking the web pages. Smoke tests ensure that all the components and services collaborate together and the whole product behaves as expected. They are the last check before the product & service is delivered to customers. Therefore, it is a crucial piece in the overall testing strategy.

The Hootsuite dashboard has complex features that are highly interactive and work across multiple platforms. Choosing the right tool and writing smoke tests that are efficient, highly automatic, and easy to maintain is by no means a simple task. Through trial and error, Hootsuite has applied several popular tools to implement our own smoke tests architecture and finalized what we call the “Venkman” testing framework, which is built on top of Selenium and NightwatchJS. But first, we’ll start with discussing our previous PhantomJS framework, internally referred as “Casper”.

Our previous tool — “Casper” (PhantomJS + CasperJS)

  • What it is?
Our “Casper” framework is built on PhantomJS and CasperJS. PhantomJS is a headless WebKit browser that is scriptable with a JavaScript API. To put it in a simpler way, it is a browser without GUI, but rather, using command line interfaces to automate the control of a web page. PhantomJS is light and fast and handy to use and it directly runs on the command-line. You can easily add it into the git precommit hook. So it is a great tool for fast development workflow. CasperJS is a functional testing framework for PhantomJS. It eases the process of defining a full navigation scenario and provides high-level functions and syntactic sugar for PhantomJS. In addition, it has Casper Queue to manage the event sequence, so you can use then() to chain functions. Overall, The highlight of this framework is that — it is fast and handy with minimal dependencies, which is the primary reason why Hootsuite originally chose it.

                                               

  • The pains
Abandoned by the Community — The “Casper” framework at Hootsuite has rarely been changed since its creation. Mostly because the support for both CasperJS and PhantomJS have slowed down the last couple of years. Due to this inactivity, bugs and compatibility issues came along. One of the biggest issues is that it does not support modern web frameworks like ReactJS. As our dashboard has migrated the front-end to use React components, the limited support for testing became increasingly challenging. Also updates of CasperJS and PhantomJS are not in sync, and in order to avoid version conflicts, we tried to lock our version to a specific one, which caused a great amount of confusion and affected productivity.

Hard to write — By the nature of JavaScript, functions in CasperJS are asynchronous, which makes it tricky to write tests where order is important. On top of that, Hootsuite as an organization changed its QA team structure. The previous structure utilized an embedded and specialized QA within each team. The new structure has a centralized QA team that builds and monitors the testing workflow while the respective developers took on the QA role themselves and started writing their own tests. After the change, it was a big challenge to train all developers on writing good “Casper” tests.

Limited Test Accounts — This problem is not associated with the CasperJS framework, but actually related to how we have designed our test suite. Mocking up the scenario is one of the most complicated steps in end-to-end testing. At Hootsuite, we need to setup test social media accounts for each test. Previously, only a few hard-coded testing accounts were available in pipeline. Tests running in parallel had to compete for use of those limited test accounts, causing the tests to be unstable. For example, each Hootsuite account is restricted to a maximum of three active sessions. This leads to problems when a batch of tests are running against these hard-coded testing accounts. Some tests will fail simply because new tests authenticate using the same account.

                         After spending a huge amount of time investigating and customizing the CasperJS library, we realized that maintaining this framework took more effort than switching to a new testing framework. By experimenting, we found that on average, CasperJS was not as fast as we initially thought. Here is an example of our experiment: we randomly chose a test suite and wrote it for both CasperJS (on PhantomJS) and NightwatchJS (on Selenium), ran both tests ten times and compared the running time. (The result is shown in the graphs above) The performance difference between the two approaches is not as much as we thought. Overall, considering all of these pains and few gains, we finally made our decision to deprecate our Hootsuite “Casper” framework.

Our new solution — “Venkman” (Selenium + NightwatchJS)

To end the Casper’s nightmare, Hootsuite built up a new internal testing framework with a nick name “Venkman”.  “Venkman” is an in-house UI testing framework based on NightwatchJS that is driven by Selenium. Selenium (specifically Selenium 2) is a widely used browser automation tool for writing functional tests of web applications. Speaking from a high level, Selenium 2 is a combination of the Selenium Core which is written in JavaScript to control the browser, and WebDriver which provides language bindings and each individual browser control. It supports a number of languages (JS, Java, C#, Python etc.) as well as most commonly used browsers. At Hootsuite we use JavaScript, namely NightwatchJS to write tests that runs on Selenium. NightwatchJS is a automated testing framework written in node.js. We use it simply because it also uses an owl as its brand icon… No, I’m kidding. We adopt it because it has simple syntax, making automated testing easy to write and read. Moreover, it has clear BDD-style assertions.

                                   

Other than the testing language and framework, we have adopted new technologies such as Docker and Consul to build a more robust architecture of the smoke testing system. Integrating them all together, we have gained tremendous benefits:

  • The gains
Comprehensive Test Accounts System — As mentioned before, the previous test accounts system is too simplistic and unstable. To tackle this problem, Hootsuite built a more comprehensive test account system which is composed of two sub-services — “hsapi” and Consul key-value storage. “Hsapi” is an internal library that provides API endpoints so that you can dynamically create and teardown Hootsuite users/organizations/social profiles, which are commonly used in tests. This not only simplifies and speeds up the setup process for each test suite, but also separates the test account mock-up from the core test logic so that the tests are easier to maintain. Writing tests becomes considerably easier, for example, creating a twitter account is just the following

The other part of the system deals with the social network accounts. Since we cannot dynamically create and destroy fake accounts in social networks like Twitter and Facebook, a big pool of pre-generated social network accounts are created for each dev/staging/production environment. Those accounts are stored and managed by Consul. A good thing about Consul is that it has built-in locking system to concurrently manage the accounts. When a test runs and requires a testing account in the Consul database, Consul will lock this account and release it when the test is done. The new test account system massively improved the stability, scalability, and maintainability of our tests.

Scalable Docker container — As our “Venkman” framework integrates many developer tools, setting up the environments of the set of tools for each developer is a big challenge. Docker container is a great solution for this issue. In a nutshell, Docker is a lightweight virtual machine that guarantees the software installed in the container will always run the same, regardless of its environment. For “Venkman”, we use three docker containers to run the three most important components simultaneously — node.js server for Nightwatch.js; Selenium and browsers (Chrome and Firefox) that are tested against; and the Consul agent mentioned before. To set all of these up, requires only two commands. The adoption of Docker successfully avoids common pains and allows developers to focus on writing tests rather than fixing bugs in environment settings.

Clear Assertion Style — NightwatchJS uses Behavior Driven Development assertion style, making both the test codes and test reports easy to understand. And as the syntax is very straightforward, the learning curve is very short. These features makes NightwatchJS a suitable tool for Hootsuite where every developer needs to write their own smoke tests.

Running Against Real Browsers — Thanks to the Selenium framework, our tests are able to directly test against real browsers, like Chrome and Firefox. The tests are much more close to the actual user experience, which gives us more confidence about the reliability of the smoke tests. In addition, besides taking screenshots while the tests are running, the tester can even watch the tests running in the real browser and interact with it via a browser extensions like VNC. This makes detecting the reasons of failed tests much easier as we can see what is actually happening.

Conclusion

Talking about the pros and cons of our “Casper” and “Venkman” frameworks, my point is not to convince you that one is better than the other. There is no generic tool that will work for all businesses and all situations. As your project scales or your environment changes, you may need to keep up the pace by changing your tool kit accordingly. And since no one can tell you what tool will be the best fit, the only way to figure it out is to always be open and explore new technologies. This is exactly what has happened here at Hootsuite, and how we dealt with it. In the future, the improvement to “Venkman” and exploration to new end-to-end testing frameworks will go on at Hootsuite as the current “Venkman” is still on its way to a mature testing framework. Our journey continues!

Thanks to @Shane Gearon for sharing the history of our testing frameworks. Additional thanks to @Paul Cowles and @Adam Arsenault and @Martin Jung for polishing this post.

 

About the Author

Tom Yang is a Co-op software developer. He studies Business and Computer Science at UBC. He enjoys front-end development and loves learning various web developing technologies. Besides programming, he is also passionate about playing billiards and hiking. You can find more about Tom here.