How to make your automated tests run faster with Bazel
For our continuous delivery practice, an important part of shipping code to production is running automated tests against those production candidates to ensure the version of those applications work as we expect. Sometimes those automated tests take so long to run that it feels like it kills the flow of continually shipping. So, one of our goals this summer was to reduce testing time, even when the number of tests is high, so we can keep pushing code to production. By introducing Bazel to our Insights product we were able to reduce the testing time by 50% from 7 minutes to 3.5 minutes, and reduced our time-to-deploy by 70% from 9 minutes to 2.7 minutes.
BazelBazel is the open-sourced cousin of Google’s internal tool – Blaze. Bazel allows the automation of building and testing, developed by Google too. What makes Blaze unique is that it has been designed to solve build problems specific to Google’s development environment. According to Jeff Cox from Google’s Bazel team, it is suitable for projects that involve massive, shared code repositories, extensive testing and release processes and language and platform diversity.
It has proven to be both scalable and reliable after extensive usage by tens of thousands of engineers and tested over years on Google’s environment. It has proven to be build software quickly and correctly too. It’s speed is achieved by caching and parallel execution, as Bazel rebuilds only the files that are needed, instead of the entire project and runs tests only for the code that has been changed.
To understand Bazel and see if we could benefit from it, we first had to understand how it worked. Bazel works with WORKSPACES and each workspace is defined by a file named WORKSPACE (which can be empty). Inside every workspace you can define BUILD files with rules and each defined rule is a target. Each target will be a node in the dependency graph, and the target’s dependencies are edges.
Rules describe the dependencies between files and packages. If you want to get an analysis of these dependencies, you can use the query language offered by Bazel. Bazel’s query language is a language of expressions which operate over the build dependency graph, which is the graph implicitly defined by all rule declarations in all BUILD files. You can find the dependencies of a rule, trace the dependency chain between two packages or select the set of targets that depend on some target by executing a Bazel query.
For example, if you change a library and you want to find what are the dependent libraries, you can run the following query: bazel query ‘kind(library, rdeps(//…, //database/mongodb:mongodb))’, and you can also visualize the graph of your dependencies”
TestsBazel offers possibilities for testing and one of them is writing build rules that define automatic tests which can be run with ‘bazel test … ’ command. The tests runner can work on multiple test targets defined by rules. Bazel provides an parallelization system which allows multiple jobs at the same time to be executed. The system can used by specifying –jobs argument to ‘bazel test’ command. Another built-in system which comes with Bazel is it’s Caching System. It caches all PASSED tests, and when the same tests are executed, if no change was made to files or dependencies, they will be skipped. This is the main reason we used Bazel in our project and we have an improvement of 50% on the time spent on testing from 5 minutes to 2.5 minutes.
Another strong feature of testing in Bazel is specifying a size or time for each test. By using it you can establish for tests time limit, and if tests took more than that, it will automatically fail. More options for testing can be found on official tests documentation.
Extending rulesBazel offers flexibility on rules and allows you to write your own rules using Skylark Language and import them in BUILD files, and all targets defined with a custom rule will be executed with custom rule’s code. We use this possibility to write our custom rule of py_test, because the default rule is using pytest as tests runner, and we needed nosetests.
This extension is possible because every rule is converted in a bash script which is executed inside inner tree of symlinks.
ResultsAfter integrated a big part of project with Bazel we have obtained next results:
In 34.5% of cases from a metric of 500 tests, only 100 to 200 tests are executed. This means almost one third of total tests are executed so testing time can decrease by more than half (depending on execution time and dependencies). There are many chances for a test to be executed if it has a big number of dependencies.
So, when we hit the deploy button, only 23 % of tests are executed and total time spent on deploy was reduced from 9 minutes to 2.7 minutes, and a considerable part of the time is deploy’s overhead, which it cannot be diminished with Bazel.
ConclusionGiven that Bazel is developed and supported by Google with the stated goal of productive development environment, where code can be shipped as fast and bug-free as possible, we can trust the reliability and flexibility of this build system.
If your project has a large shared codebase that supports multiple platform, written in multiple languages which has a huge number of tests, and if you desire a full overview of base code, Bazel might be a good choice as a build system for it.
About the Authors
Teodor and Rodica are students in 4th year at University of Politehnica Bucharest at Faculty of Automatic Control and Computer Science where we found our passion for technology and innovation. We’ve been lucky to work this summer with Hootsuite team from Bucharest, as interns and to help them to improve testing time on deploys. Beside technology, Teodor loves basketball and to travel as much as he can, meanwhile Rodica likes to read novels and finds a passion in everything she does. But now, as the summer has ended, it’s time for us to get back to school and start working on our final projects for bachelor’s degrees.