Our Journey In Smoke Test Frameworks
Smoke testing is an important part in Hootsuite’s Build Pipeline. During my work term at Hootsuite, I got a chance to be a part of the change from a PhantomJS based framework to a Selenium one. The purpose of this post is to share my learnings from this journey — and that there is no so-called ‘best’ framework that will work in any situation. Choose the right tool for your business.
Overview of Web Smoke TestsThe smoke test is under the category of end-to-end tests which inspect the basic functionalities of a project. It is similar to the functional test but not as detailed.
In the practice with web applications, the standard way is to automate a browser (either a headless one like PhantomJS or a real one like Chrome/Firefox) and simulate the actions as if a user is actually viewing and clicking the web pages. Smoke tests ensure that all the components and services collaborate together and the whole product behaves as expected. They are the last check before the product & service is delivered to customers. Therefore, it is a crucial piece in the overall testing strategy.
The Hootsuite dashboard has complex features that are highly interactive and work across multiple platforms. Choosing the right tool and writing smoke tests that are efficient, highly automatic, and easy to maintain is by no means a simple task. Through trial and error, Hootsuite has applied several popular tools to implement our own smoke tests architecture and finalized what we call the “Venkman” testing framework, which is built on top of Selenium and NightwatchJS. But first, we’ll start with discussing our previous PhantomJS framework, internally referred as “Casper”.
Our previous tool — “Casper” (PhantomJS + CasperJS)
- What it is?
- The pains
Limited Test Accounts — This problem is not associated with the CasperJS framework, but actually related to how we have designed our test suite. Mocking up the scenario is one of the most complicated steps in end-to-end testing. At Hootsuite, we need to setup test social media accounts for each test. Previously, only a few hard-coded testing accounts were available in pipeline. Tests running in parallel had to compete for use of those limited test accounts, causing the tests to be unstable. For example, each Hootsuite account is restricted to a maximum of three active sessions. This leads to problems when a batch of tests are running against these hard-coded testing accounts. Some tests will fail simply because new tests authenticate using the same account.
After spending a huge amount of time investigating and customizing the CasperJS library, we realized that maintaining this framework took more effort than switching to a new testing framework. By experimenting, we found that on average, CasperJS was not as fast as we initially thought. Here is an example of our experiment: we randomly chose a test suite and wrote it for both CasperJS (on PhantomJS) and NightwatchJS (on Selenium), ran both tests ten times and compared the running time. (The result is shown in the graphs above) The performance difference between the two approaches is not as much as we thought. Overall, considering all of these pains and few gains, we finally made our decision to deprecate our Hootsuite “Casper” framework.
Other than the testing language and framework, we have adopted new technologies such as Docker and Consul to build a more robust architecture of the smoke testing system. Integrating them all together, we have gained tremendous benefits:
- The gains
The other part of the system deals with the social network accounts. Since we cannot dynamically create and destroy fake accounts in social networks like Twitter and Facebook, a big pool of pre-generated social network accounts are created for each dev/staging/production environment. Those accounts are stored and managed by Consul. A good thing about Consul is that it has built-in locking system to concurrently manage the accounts. When a test runs and requires a testing account in the Consul database, Consul will lock this account and release it when the test is done. The new test account system massively improved the stability, scalability, and maintainability of our tests.
Scalable Docker container — As our “Venkman” framework integrates many developer tools, setting up the environments of the set of tools for each developer is a big challenge. Docker container is a great solution for this issue. In a nutshell, Docker is a lightweight virtual machine that guarantees the software installed in the container will always run the same, regardless of its environment. For “Venkman”, we use three docker containers to run the three most important components simultaneously — node.js server for Nightwatch.js; Selenium and browsers (Chrome and Firefox) that are tested against; and the Consul agent mentioned before. To set all of these up, requires only two commands. The adoption of Docker successfully avoids common pains and allows developers to focus on writing tests rather than fixing bugs in environment settings.
Clear Assertion Style — NightwatchJS uses Behavior Driven Development assertion style, making both the test codes and test reports easy to understand. And as the syntax is very straightforward, the learning curve is very short. These features makes NightwatchJS a suitable tool for Hootsuite where every developer needs to write their own smoke tests.
Running Against Real Browsers — Thanks to the Selenium framework, our tests are able to directly test against real browsers, like Chrome and Firefox. The tests are much more close to the actual user experience, which gives us more confidence about the reliability of the smoke tests. In addition, besides taking screenshots while the tests are running, the tester can even watch the tests running in the real browser and interact with it via a browser extensions like VNC. This makes detecting the reasons of failed tests much easier as we can see what is actually happening.
ConclusionTalking about the pros and cons of our “Casper” and “Venkman” frameworks, my point is not to convince you that one is better than the other. There is no generic tool that will work for all businesses and all situations. As your project scales or your environment changes, you may need to keep up the pace by changing your tool kit accordingly. And since no one can tell you what tool will be the best fit, the only way to figure it out is to always be open and explore new technologies. This is exactly what has happened here at Hootsuite, and how we dealt with it. In the future, the improvement to “Venkman” and exploration to new end-to-end testing frameworks will go on at Hootsuite as the current “Venkman” is still on its way to a mature testing framework. Our journey continues!
About the AuthorTom Yang is a Co-op software developer. He studies Business and Computer Science at UBC. He enjoys front-end development and loves learning various web developing technologies. Besides programming, he is also passionate about playing billiards and hiking. You can find more about Tom here.