Test Automation (visual regression test) for UI comparison on web applications using BackstopJS.

Test Automation (visual regression test) for UI comparison on web applications using BackstopJS.

Before I get into helping you set up this application, please allow me to explain a simple scenario at work where this solution was useful and how it made my life easier. PS: I am not a QA Engineer although I thought this could be something that could be introduced in any web project or application as a testing automation process when it comes to visual regression.

When and How this occurred ?

It was the usual deployment day and a very strange issue took place, where some of the frontend UI changes in the UAT / QA environment was not being reflected on the Production / Live environment. These kind of issues are hard to be spotted on the day of deployment, as there are higher priority features that need to be tested. Although, when we rectified some logos on the site were moved to the right, some images were not correctly indented and so on.. Nightmare! The first thing our developer checked was, to compare the CSS files in the Chrome console that were on the UAT / QA server with the Live / Production server- and there it was .. the CSS classes were missing and some did not have the required changes. :(


After some research online, I did find one such tool that would avoid such a situation. It would scan the same pages from different environments and point out what are the differences at a CSS and pixel level. This was none other than BackstopJS.

What is BackstopJS OR Visual Regression Testing?

BackstopJS is an open source automation developed by Garris and a bunch of other clever engineers. It automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.

Impatient ? Let me show you what the end result looks like ;)

These are three screenshots from three different environments or URLs. Highlighted in pink are the ‘changes in UI’ that the tool detects. Interesting ?lets go ahead and set it up.

Installing nodejs and npm — dependencies for BackstopJS

We have multiple ways to install BackstopJS, lets use npm in our case. What is npm ? Read here.. https://docs.npmjs.com/about-npm

In short, what is npm ? Its a Node package and dependency manager. There are millions of packages out there (imagine a library with many books), one such package is backstop JS. Although lets start from step 0.

To install npm you need a pre-requisite of NodeJS in your machine. Download the LTS version from here https://nodejs.org/en/download/ (you can choose Windows or Mac or Linux depending on your machine OS)

Once NodeJS is installed successfully you can verify by running this command in your terminal / Command Prompt. Lets begin..

  1. Windows > Start > CMD (better to right click and open as Administrator)
  2. type node node -v OR node --version > hit enter > below is what you should see

3. Install npm, npm install npm@latest -g this will install the latest version of npm on your machine.

4. Verify npm by typing npm -v in CMD same as step 2, below is what you should see

Now you have successfully installed NodeJS + npm on your machine.

Installing backstop, finally! :D

  1. Open terminal / command prompt and type npm install -g backstopjs (here g means, we are installing backstopjs globally and can be run through any part of the system) below is what you should see in your terminal as a result of running this command

2. As you see in the console / cmd along with backstopjs the package also installs a ‘headless chrome browser called chromium’. BackstopJS uses this headless browser to load the web pages and then compares them individually which is the final output for us.

Code Editor

  1. Before initializing BackstopJS I would highly recommend you to install any editor of your choice, although my preference would be Visual Studio Code. It is free, light, has intellisense to autofill your code, error handling and also supports almost all coding languages(this way you don’t need to switch windows from an editor to cmd) Download from here https://code.visualstudio.com/Download
  2. Create a new folder in your machine and open it in visual studio code. For this open visual studio code > file > open folder > locate the folder you created. In my case I have created a backstop_medium and this is how it looks, its a blank canvas :)

2. In the top bar click on Terminal > New Terminal > below is what you should see in the bottom window.

3. Lets run those commands !

Initialize BackstopJS

  1. In the terminal we will type backstop init hit enter.

There are two changes that you can see in your editor now,

  1. A successful initialization of backstop.
  2. A new folder that is created under your main folder. backstop_data
  3. A {}backstop.json file that is created separately. For now lets focus on this file in particular as it help us add configurations to run our test.

The backstop.json file

  1. The backstop.json file is the control room for the configurations you enter and based on these configurations the tool performs its comparison.

Viewports are the screen sizes in which you want the browser to compare your website. This helps when your website is responsive and you can compare the screens at a mobile, tablet, iPad and desktop level.

You may keep adding viewports in the above format specifying the sizes of the screens that you want to compare and define a label for each of the screens.

2. Scenarios — here you can define multiple actions that backstop needs to consider- such as:

In the above screenshot you can see a “url” which is the live url you want to test and referenceURL is that you want to compare with. In our case we are testing + comparing the uat.website.com (test) with the website.com (Live working website). Other labels can be interpreted in the below definitions

Puppeteer mentioned above is another kind of a headless browser with additional capabilities than chromium. In case you want to stick to Chromium, change “engine”: “puppeteer” to engine : “chromium”

You can install puppeteer using a npm command. Simply in your terminal type: npm i puppeteer

PS: I can explain any specific scenario in detail if you wish to, please mention in comments. In this article I am explaining a basic test run for any two pages / URLs. You can add multiple scenarios, set a delay time and cover more test cases for visual regression.

Run the test

After all your settings are in place- remember to save the json file, post that in the terminal you can run backstop test . You will see various actions being processed in your terminal and your browser will automatically open your result html page with pass / fail, screenshots and more.. You can even click on any one particular file and compare each section of the screen using a scaler.

Feel free to comment in the comments below.

Product Management, building better user experiences by understanding personas, passion for design , Tech enthusiast and musician. Always learning..