How to Perform Visual Regression Testing Using Playwright

Regression testing verifies that system changes do not interfere with existing features or code structure. They are part of almost every test suite in software development lifecycles. It is common for developers to change or add a code section and unintentionally disrupt something that is working just fine.

Visual regression testing functions on the same logic but confines it to the visual aspects of the software. It works by comparing two images and automating complicated scenarios, like when we cannot identify the elements in the DOM tree. However, visual regression can be used on any website.

How Does Visual Regression Testing Work?

During the first run, the visual comparison tool captures the snapshot called the base image. The subsequent run compares the base image if there is no difference test is passed, and if there is a difference, the test is considered as failed. Visual regression is also called visual comparison testing.

In this tutorial, we will discuss automated visual regression using Playwright.

Pre-requisites for Visual Regression with Playwright

• Download and install NodeJS
• Download and install Visual Studio Code (Recommended)
• Install Playwright NPM module
• Install @playwright/test module

Note

1. Throughout this tutorial, we are using Playwright with JavaScript.
2. Playwright comes with the default visual comparison tool, so there is no need to install additional packages.

Create Simple Visual Comparison Tests Using Playwright

In your tests folder, create a new JavaScript file example demo.spec.jspage.screenshot() function takes the screenshot, and expect in the @playwright/test module provides the assertion for matching the images that are .toMatchSnapshot().

Inside the newly created JavaScript file, create a new test that performs the visual comparison like below.

// example.spec.js
const { test, expect } = require('@playwright/test');
test('Visual Comparison Test Demo', async ({ page }) => {
  await page.goto('https://playwright.dev');
  expect(await page.screenshot()).toMatchSnapshot();
});

In the above code, we navigate to the official Playwright website and compare visually by taking a snapshot.

Run Your First Visual Regression Tests

Run your visual comparison tests using the below command:

npx playwright test

The first time the test is run, it fails, as there is no base image.

As seen below, the directory containing the base image file gets created inside the tests folder.

Run the spec again; it passes.

Visual Comparison in Playwright to Ignore Minor Differences

The above comparison technique matches the screenshot pixel by pixel, which means each pixel should match exactly. This behavior can be modified by passing the argument maxDiffPixels = <pixel_value>.

Example

const { test, expect } = require('@playwright/test');
test('Visual Comparison Test Demo', async ({ page }) => {
  await page.goto('https://playwright.dev');
  expect(await page.screenshot()).toMatchSnapshot({ maxDiffPixels: 200 });
});

In the above example, we have specified the maxDiffPixels value as 200, which means the maximum pixel difference can be 200.

Image Comparison in Playwright With Threshold Option

Playwright toMatchSnapshot() accepts threshold, threshold ranges from 0 to 1, default value is 0.2. The threshold is tolerance of image differences.

Example Code

const { test, expect } = require('@playwright/test');
test('Visual Comparison Test Demo', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  expect(await page.screenshot()).toMatchSnapshot({threshold:0.5});
});

In the above code, the threshold is mentioned as 0.5.

Playwright Visual Comparison Tips and Tricks 

In Playwright, we can pass the image file name; instead of default comparison, Playwright compares with the specified filename.

Example

 expect(await page.screenshot()).toMatchSnapshot('home.png');

Playwright also allows us to compare element snapshots; we can take a snapshot of DOM elements and compare.

Example

expect(await page.locator('xpath=//*[@id="__docusaurus']).screenshot()).toMatchSnapshot();

Visual Regression With Playwright Using Percy

Percy is a web-based tool for visual testing with a free tier, and it provides both manual and automation capability for visual comparison. Percy supports Playwright integration. 

Percy is now a part of Browserstack. If you already have a BrowserStack account, you can sign in with BrowserStack or sign up and create one.

Using Percy With Playwright

Step 1 – Install Percy modules using the following command.

npm install --save-dev @percy/cli @percy/playwright

Step 2 – Create a new JavaScript Playwright test file like below.

//demo.spec.ts
const { chromium } = require('playwright');
const percySnapshot = require('@percy/playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://www.browserstack.com/', { waitUntil: 'networkidle' });
  await percySnapshot(page, 'Example Site');
  await browser.close();
})();

In the above example, we are navigating to https://www.browserstack.com/, and we are taking a snapshot using the percySnapshot() function.

Setting Up Percy

Step 1 – Login to Percy. If you don’t have an account, create one.

Step 2 – Create a new project.

Step 3 – Copy Percy token.

Step 4 – In your Visual Studio Code Terminal, set the PERCY_TOKEN environment variable using the below commands:

Powershell / Visual Studio Code Terminal

$env:PERCY_TOKEN = "your_token"

Windows Command Line

set PERCY_TOKEN="your_token"

Unix/Linux Based OS

export PERCY_TOKEN="your_token"

Step 5 – Run your tests using the below command:

npx percy exec -- node tests\demo.spec.js

Note: tests\demo.spec.js is our test file name. Provide the relative path of your test file name.

Step 6 – Start Execution and wait until it finishes.

Once Percy completes the tests, it shows in the console log.

Console Log provides the build URL. Once you navigate to that URL, you can see your image uploaded into  Percy.

If there is no difference in the image, it says “No Changes,” like below.

If there is an image difference, it will show the difference side by side like below.