Step-by-Step VRT with Mugshot

During my co-op, I was given a specific task: implement a visual test suite to check that the parts that make up our Analytics app don’t fall prey to unexpected styling changes. In this article, I will be talking about what tools I used, the contributions I made to them, and what challenges I had to overcome.

What are visual regression tests?

If you are a newbie to testing, I advise you to carefully read this article that one of our previous co-ops wrote.

In short, every functionality of a product should be covered in tests. Mainly, there are three types of tests:

  • Unit: covers small parts of the app logic
  • Integration: checks the interoperability between those parts
  • End-to-End: tests the full functionality of the app
image07

With these in place, we’re sure that our product is up and running and no developer can break the code. But what about the style? Visual regression tests ensure that the way a product looks does not regress over time.

This requires running the app and taking “pictures” of the components that build it. This process has to be run at every change in the code, and each time the image must be compared to the one made before. That is how you make sure that nothing unexpected has changed, like one very liberal CSS selector that affects components that we’re not aware of.

Why do we need them?

The usual process is to periodically check the app by hand. Manual checking exclusively is time-consuming, expensive, and limited. Also, for the human eye, it’s pretty hard to notice when something is a few pixels different. You may say that some pixels don’t make such a big difference. Indeed, but what happens when you have layouts that depend on fixed sizes?

For example, take a look at these two images side by side and see if you can spot any changes.

This is an example page that I took from the Bootstrap repository.

click to view larger
click to view larger

click to view larger
click to view larger
While the most striking differences can be easy to spot, a non-biased artificial eye could find changes that we might omit. Here’s what I mean:

image08

Where to Start

You can start by thinking about what you can really test. For example, you can not test a data visualization widget that will display new values every day. You can only test the HTML elements that are not supposed to change over time.

A solution that my team found is to build a package with the components used in our Analytics application: the buttons, dropdowns, icons, search bars, etc. We call this package HS-Components. The components are mainly build using React. They have some built-in characteristics but you can also define the way they look or behave through props. We achieved this using React Cosmos. You can find what is it and how it works here.

Using our components package, when we say something like: “I want a button that is usually used for confirmation and I want it to have the text ‘Submit'”. HS-Components will know that the button has to be green, a little rounded at the corners, have some padding, and the text required.

The goal is to have HS-Components include all elements used in Analytics, and to build both Analytics and Insights upon that package. Like this, all of the components are isolated and we can render them in different use cases and run visual tests on them.

Useful tools and code snippets

All the packages mentioned in the following run on Node.js.

MOCHA

Mocha is a JavaScript test framework, that makes asynchronous testing simple and fun. It allows a natural description of the tests.

You tell mocha what to test: 

1
mocha test.js

and the output will look like this:

image03

The example above is pretty simple, but let’s say that instead of an array or string you have a function call. Then you can write an ‘expect’ statement with what that function should return for a given input.

CHAI

In the previous example, we used 

1
expect

from Chai.

Chai is an Assertion Library that can be paired with any javascript testing framework. It has several interfaces that allow the developer to choose the one he/she is most comfortable with.

Chai usage
Chai usage

MUGSHOT

Mugshot is a library for visual regression testing. It helps you identify the parts of your product that have unexpected visual changes.

The beauty of Mugshot comes from its independence of any browser, test-runner, diffing tool, etc. It achieves this through interfaces over these dependencies and comes with some implementations of its own.

To run Mugshot you will need:

  • A tool that checks for the differences between two images (I recommend Looks Same)
  • A web driver, which is a tool for automating web application testing that lets you control a browser. You can access and manipulate the DOM of the requested web page. I suggest WebDriverIO.
  • A browser to render the web page. PhantomJS is native, fast, and perfect for basic testing. Unfortunately, it uses an old WebKit version, but if you need the latest versions of popular browsers they’re available in Firefox or Chrome.
  • A server to serve the web page. If you use PhantomJS as a browser, you should pair it with Phantomjs-Prebuilt as a server. If you need Firefox or Chrome, Selenium-Standalone has the easiest setup.
  • Adapters for the chosen web driver and diff tool. Mugshot-WebdriverIO is the adapter that we build for WebDriverIO and Mugshot-Looks-Same for Looks Same. Looks Same is Mugshot’s default diff tool, and must be installed if you don’t provide another one. We will talk about those adapters later on but until then you can take them as they are.

The flow is really simple. Provide Mugshot three things: a browser instance, an HTML element, and a baseline image, then tell it to do the testing. The result is an image with highlighted differences.

What's different?
What’s different?

More about the Mugshot API can be found here.

A step-by-step example of Visual Regression Testing

Let’s do some testing! As in the previous example, we’ll use the Chai assertion library as well as WebDriverIO for creating a browser instance and its Mugshot adapter.

We’re going to be using Chrome and it will be run through a Selenium server. The Selenium server should be running in the background. The code loads the required web page and we then give Mugshot the browser instance so it can access the DOM.

Mugshot has one method:

1
test

. In the following example, it will test the HTML Element with the class

1
button-primary

and will generate diffs based on

1
green-button.png

baseline image. Here, we expect no differences.

You can visual test only by using the tools mentioned up to now: Mocha, Chai and Mugshot. If you want a better understanding of the output you should use Chai-Mugshot and Mocha-Mugshot-Reporter that we will discuss later in this post.

In the following, we will discuss the two adapters for Mugshot. You don’t really have to understand how they work in order to use Mugshot, but it will definitely help if you want to write your own adapter for your favorite tool. Here is a good start for understanding Mugshot’s adapters.

MUGSHOT-WEBDRIVERIO

We decided that WebDriverIO is one of the best web drivers, and we built an adapter over it.

Let’s have a sneak peek in the browser interface. It’s easy to see that we’re going to need two methods:

1
takeScreenshot

and

1
getBoundingClientRect

.

The first method has to return the data through a buffer. The data represents the screenshot image. Taking a look at the WebDriverIO API we see that it has a method like this called

1
SaveScreenshot

. Awesome! All that we have to do now in our adapter is to call that function:

We explore more the WebDriverIO API but we can not find a method that provides an element’s coordinates. But we do have

1
getElementSize

to find out the element size and

1
getLocation

to find its location on the page. Then we can calculate the coordinates.

You can find the full source file here.

MUGSHOT-LOOKS-SAME

Looks Same is a library for comparing PNG-images, taking into account human color perception.

Let’s take a look into the differ interface. We see that Mugshot requires two methods: 

1
isEqual

and

1
createDiff

. We adapt the Looks Same methods to what Mugshot needs:

Here you can find the source file.

CHAI-MUGSHOT

Let’s take the example in the Mugshot subchapter, and suppose that the test fails.

Because of the assertion:

1
expect(result.Equal).to.be.true

, the output will look something like in the following image. That is not really helpful when you have 100 tests or more.

image10

For a better understanding of which component regressed, we created the Chai-Mugshot plugin.

For the plugin to function, you must pass a Mugshot instance:

image12

Then you just say what you expect, no need for the actual call of the

1
test

function. The plugin will do that for you.

image00

As you can see, the output is so much more reliable providing the baseline name and the screen selector.

image13

You can find a complete example here.

MOCHA-MUGSHOT-REPORTER

Mocha reporters are plugins that display the mocha output in a more friendly way. We wanted to build a dedicated interface for visual regression tests. Mocha-Mugshot-Reporter is just that.

Usage:

mocha test.js -R mocha-mugshot-reporter

In the code, you need to hand out the test context to the assertion library.

image12

It will create a runner that displays the diffs in a GitHub style. It is easier to check for differences side by side and you have multiple comparison modes. Little changes are hard to spot by a human eye.

Another plus is that when you intentionally want to change a component, you can send the generated report to the design team for confirmation. They can directly identify the changes, without having to set up any development environment.

image05

DOCKER

Docker is an integrated, easy-to-deploy environment for building, assembling, and shipping applications. It’s like a virtual machine that can help on doing your work while waiting for the tests to run. Also, you can install a certain image of the browser and always do the testing on that so you will not have false positives due to updates. Another tool that I recommend is Kitematic. It provides the output of the docker containers in an app.

GIT LFS

One of the things I had to figure out was where to keep the baselines for the visual tests. We need to access them easily, have some kind of integration with Git and a history system. The most obvious solution is to keep the baselines in the repo. That will increase clone times for our repo as all of the versions of each image are part of the history.

The best solution that we found was Git Large File Storage. It replaces large files with text pointers inside Git, while storing the file contents on a remote server. The setup is easy and you only need to make it once. You tell what files to track and from that point, Git will know to put them in Git LFS. As for the user experience, it remains the same. You can pull, commit, push as you always did.

Possible Challenges

Mocha Timeout

If you use other browsers than PhantomJS, you may need to extend Mocha’s default timeout. You can do this:

  • at runtime:  

    1
    mocha test.js --timeout 6000

  • in the code:

    1
    this.timeout(6000)

    where

    1
    this

    is the suite for which you want to extend the timeout

Wait until the page is loaded

Web drivers don’t always wait until the page is fully loaded before handing out the browser instance. One of the things I researched is how I  can wait for a page to load before I do the testing. WebDriverIO has a method called

1
executeAsync

that provides the browser environment. You have access to

1
document

and

1
window

objects so there are countless ways to wait until the state you desire.

The most common way is to wait until

1
document.readyState

becomes

1
complete

. Somehow, in Chrome, it happens that even if the state is complete, fonts are still not loaded.

A useful hack that I found is to check when

1
document.fonts.status

is loaded and then do the testing. This property of the

1
document

object exists only in Chrome.

Conclusion

I hope this article will help you take the first steps in visual testing. If you want to see a full working example of Mugshot, you can check out the Mugshot Demo repo.

All the packages I mentioned are open source and you can contribute to them. Part of my job was to improve those packages and make them fit our needs. You can find more about those contributions here.

The future brings more challenges. One of them is to integrate Mugshot in the main Analytics repo and continue working on the visual test flow so that it’s as seamless as possible.

About the Author
Adela Istrate

Adela is a Co-op on Hootsuite’s Analytics UI team. She is eager to learn and not afraid of new challenges. She enjoys web development and everything that has to do with manipulating the DOM. Board Games are one of her favorite things to do in her free time.