Developer Guide: How to catch visual bugs using Playwright

By now, those working in software development know what functional tests and end-to-end tests are. They also know test automation is the key to speeding up their development process without compromising software quality.  

However, visual testing, a set of tests designed to make sure that your app “looks” as intended, is an often-forgotten facet of testing. The good news here is it is easy to start visual testing, especially if you have some pre-existing functional/e2e test scripts.  

What is Automated Visual Testing? 

Automated visual testing, in practice, is an extremely simple concept.  

Let’s say you have an automated test that navigates to a website, clicks a sign-up button, and fills out a form to create a new user.  During this automated workflow, you may care about how the app looks. This is where SmartBear tool, VisualTest, can help. 

For instance, you might want to verify the visual layout of the UI as soon as a user navigates to the website. You’d want to do this to make sure the sign-up button is visible and placed in an easily click-able location (not to mention its color, drop shadow, width, height, luminescence, text content, font, font color, etc.). You might also care about the sign-up form, where you want to make sure all the form fields are presented in the right order.  

During that time in the workflow, you would add commands to take a screenshot and automatically send them to VisualTest. Upon subsequent runs of the same automated test (whether you run it again manually, or by having the test integrated with your CICD system), the command would once again take a screenshot at the same moment in time of the workflow and send the screenshot to VisualTest again. VisualTest then compares the two screenshots using our proprietary image comparison algorithm, and alerts via console logs whether any visual changes have occurred to that screen in question.  

But how does this look in practice? We’ve put together some examples below to show you. VisualTest is compatible with multiple frameworks, but for the examples shown below, we will predominantly be working in JavaScript/TypeScript using Playwright. 

Why We Chose Playwright for These Examples. 

We chose to use Playwright for these examples because it is one of the most approachable frameworks that require scripting. While some technical acumen is helpful, it’s not a must have skill as the Playwright test scripts are in English, come in highly readable syntax, and can be written with a built-in test recorder (courtesy of Microsoft’s Visual Studio Code extension).  

Playwright’s benefits don’t just stop at democratizing test automation by offering non-technical folks a way via the test recorder. As one of the newest functional testing frameworks, it is also fast and reliable.   

Users no longer need to worry about handling delays and wait times, nor do they have to scroll to elements on screen, Playwright does this automatically. Even objects within an iframe are easily accessible to the users when writing tests in Playwright. All these benefits mean more stable and robust tests. Essentially, if there is a test failure, you can be pretty sure it wasn’t because the test was flaky, but the failure is due to an actual bug.  

The Benefit of Using Playwright for Your Tests 

Playwright is a Node.js library to automate Chromium, Firefox, WebKit with a single API. This is powerful because it enables cross browser web automation out of the box. For testers, this is important because a single test can be expanded to verify all the functionalities are working as intended across all major browsers Playwright supports.  

Following the same line of thought, since all the test scripts we write can run on most supported browsers, it is also easy to start doing visual testing across those browsers as well. This will give you confidence that everything in your app looks as intended across all major browsers over time, releases, and versions. 

How to Run a Visual Test with Playwright 

Now that you have some more background, let’s build our first Playwright visual test from scratch. 

Prerequisites:

• Download Visual Studio Code 
• Download Node.js and npm 
• Start a VisualTest trial (hit the start trial button) 

If you have trouble downloading either VSCode or Node.js, then Google is your best option for debugging. It should mostly be clicking through the installation options and selecting the right operating system you’re working on. You can use the default option presented to you when you download for the first time.  

Step 1: Create a VisualTest project at https://app.visualtest.io/ 

After you create a VisualTest trial, you’ll have created a new, empty project. This will also come with a corresponding project token (highlighted in the image above) which we will use later.

Step 2: Create a New Playwright Test Suite.

First open Visual Studio Code (VSCode from here on out). Go to the extensions section in the left tab, search for ‘Playwright’ and install the one published by Microsoft.com (this is the one with hundreds of thousands of downloads). You may have to restart VSCode to reload the extension.

Go to your desktop and create a new folder. Let’s call it ‘pw_test’.

Open this folder in VSCode. It should look almost identical to the below image, where we have opened an empty folder.

Open a new terminal by clicking ‘new terminal’ under the ‘terminal’ section.

Enter the below command into the terminal:

The terminal will ask a few prompts, but you can typically start with the default options (at least for this project).

VSCode should now look like:

Now enter the below command into the terminal:

then, enter the following command into the terminal:

There should now be a new visualTest.config.js file. Take the newly generated project token from Step 1 and replace the “PROJECT_TOKEN” from the VisualTest.config.js file with the VisualTest project token from step 1.

VSCode should now look like:

Step 3: Create Your First Playwright Test

Go to the ‘Testing’ tab in the left panel and expand the Playwright section in the very bottom.

Clicking on ‘Record new’ will trigger a new chromium window to pop up. All actions you perform on the new window will be recorded as a Playwright script. So, any repetitive, tedious, time-consuming manual test can now be automated.  

For this example, let’s use https://bearstore-testsite.smartbear.com/ as a demo site to test. Navigate to the site, search for an object, then make sure that the resulting page ‘looks’ as expected (of course, you can also add in traditional assertions as you’d like).  

After you close the new chromium window that opened, you’ll end up with a brand-new test file called ‘test-1.spec.ts’ created under the ‘tests’ folder.  

Let’s add the following command to line 2: 

This import statement gives us access to the ‘sbvtCapture()’ method which will take and send the screenshots to SmartBear VisualTest.  

We want to first test that the auto-complete dropdown of my store looks ‘ok’ when a customer searches for an item. For that, let’s add a sbvtCapture command after line 7. Additionally, we’re interested in how the app looks after a customer searches for something, so let’s add another sbvtCapture command after we press enter. Then the sbvtPrintReport() method will return the image comparison results via the console logs. We’ll also add one more command to close the browser at the very end. The test-1.spec.ts now reads like this: 

import { test, expect } from '@playwright/test'; 

import {sbvtGetTestRunResult, sbvtCapture, sbvtPrintReport} from 

"@smartbear/visualtest-playwright"; 


test('test', async ({ page }) => { 

  await page.goto('https://bearstore-testsite.smartbear.com/'); 

  await page.getByPlaceholder('What are you looking for?').click(); 

  await page.getByPlaceholder('What are you looking for?').fill('ball'); 

  await sbvtCapture(page, 'searching dropdown page') 

  await page.getByPlaceholder('What are you looking for?').press('Enter'); 

  await sbvtCapture(page, 'search returned page') 

  await sbvtPrintReport() 

  await page.close() 

});    

                       

Step 4: Running Your Playwright Tests 

To run your first test, (for now), you can run tests via the VSCode UI by clicking various “play” buttons located throughout. Running the test will also run the embedded visual tests. Users will also get a link to review the visual test result if needed. For example: 

Each subsequent run of the same test will populate a new test run in SmartBear VisualTest.

Users can click into a test run to review any images that may have any found differences.

Step 5: Reviewing Your Test Results 

Let’s force an image difference to occur, so we may look over how an image review may look like. 

Change line 7, such that instead of searching for a ‘ball’, we search for a ‘watch’ instead. Line 7 should now read: 

Click on the output URL, which now shows that we have 2 image comparison failures to review. 

The images with differences will have highlights users can toggle on and off. The tool bar offers various ‘views’ to compare the two versions of the screenshots. If the found image differences are intentional changes, users can ‘accept’ the image such that it is treated as the baseline image to compare any future images against going forward. However, if the image differences are indeed defects, users can reject the image to continue using the original baseline image as the source of truth.  

If you want to expand your test coverage to various supported browser & operating system configurations, that will have to get done in the playwright.config.ts file. There is already a section called ‘projects’ within the config file that has some additional mobile browser configurations commented out. Defining default profiles for your projects to run in X, Y, or Z combinations will help you test functionalities and visual fidelity across all major browsers.  

Start using VisualTest with Playwright today!  

As you can see, getting visual tests up and running with VisualTest is simple and significantly improves the coverage you can achieve in your tests. With just a few additional lines of code, you’ll be able to start taking screenshots and automatically catch visual defects before they go out to production. Start your free trial today (no credit card required) and see for yourself! 

Free Trial