Rupert  Beatty

Rupert Beatty


Visual Regression Testing with Cypress

Visual regression testing is not a new concept. Toptal engineers routinely use it, but with looming deadlines, they sometimes need to innovate and improvise. This article demonstrates how Toptal QA specialists leveraged UI testing and Cypress to run visual regression tests without resorting to specialized tools.

Every time a new version of our component library, Picasso, is released, we update all of our front-end applications to get the most out of the new features and align our designs across all parts of our site.

Last month, we rolled out a Picasso update to the Toptal Talent Portal, the platform our talent uses to find jobs and interact with clients. Knowing the release would come with major design changes, and in an effort to minimize unexpected issues, it made sense to use visual regression testing techniques to help us find problems before the release.

Visual regression testing is not a new concept; plenty of other projects at Toptal already use it, including Picasso itself.

Tools like Percy, Happo, and Chromatic can be used to help teams build a healthy visual regression pipeline, and we did consider adding them at first. We ultimately decided the setup process would be too time-consuming and could derail our schedule. We already had a date set for a code freeze to start the migration, and with only a few days remaining until the deadline, we had no choice but to be creative.

Visual Regression Testing Through UI Testing

While we did not have visual regression tests in the project, we did have good coverage of UI integration tests using Cypress. Even though that’s not what the tool is mostly used for, Cypress has one page in its documentation dedicated to visual testing and another that lists all the available plug-ins to help configure Cypress for visual testing.

From Cypress to Screenshots

After going through the available documentation, we decided to give cypress-snapshot-plugin a try. It only took a few minutes to set up, and once we did, we quickly realized we were not in pursuit of a traditional visual regression output.

Most visual regression tools help identify unwanted changes by comparing snapshots and detecting pixel differences between a known, accepted baseline and the modified version of a page or a component. If the pixel difference is greater than a set tolerance threshold, the page or component is flagged to be examined manually. In this release, though, we knew we were going to have several small changes to most of our UI components, so setting a threshold was not applicable. Even if a given component happened to be 100% different, it might still be correct in the context of the new version. Similarly, a deviation as small as a few pixels could mean a component is not currently fit for production.


Screenshot depicting the expected result and actual result of the test run.

Figure 1. Example of minor pixel differences leading to false negatives

At that point, two contrasting things became clear: noting pixel differences was not going to help identify issues, and having a side-by-side comparison of the components was precisely what we needed. We put the snapshot plug-in aside and set out to create a collection of images with our components before and after the Picasso update was applied. That way, we could quickly scan through all the changes to determine if the new versions still matched the site’s needs and the library’s standards.

The new plan was to take a screenshot of a component, store it locally, take a new screenshot of the same component in the branch with the updated Picasso version, and then merge them into a single image. Ultimately, this new approach was not too different from what we started with, but it gave us more flexibility during the implementation phase since we no longer needed to import the plug-in and use its new commands.


Diagram showing a visual comparison flow, how images of the new and old version are merged after the visual test run.

Figure 2. Visual comparison flow


Harnessing APIs for Comparison Images

With a clear goal in mind, it was time to look at how Cypress could help us get the screenshots we needed. As mentioned, we had a good amount of UI tests covering the majority of the Talent Portal, so in an effort to collect as many critical components as possible, we decided to take screenshots of individual elements after each interaction.

An alternative approach would have been to take screenshots of the entire page at key moments during the test, but we decided those images would be too difficult to compare. Also, such comparisons could be more prone to human error, such as missing that a footer had changed.

A third option would have been to go through every single test case to decide what to capture, but that would have taken a lot more time, so sticking to all elements used on the pages seemed like a practical compromise.

We turned to Cypress’s API to generate the images. The cy.screenshot() command can, out of the box, create individual images of components, and the After Screenshot API allows us to rename files, change directories, and distinguish visual regression runs from standard ones. By combining the two, we created runs that did not affect our functional tests and enabled us to store images in their appropriate folders.

First, we extended the index.js file in our plug-ins directory to support the two new run types (baseline and comparison). Then, we set the path for our images according to the run type:

// plugins/index.js
const fs = require('fs')
const path = require('path')
module.exports = (on, config) => {
// Adding these values to your config object allows you to access them in your tests.
  config.env.baseline = process.env.BASELINE || false
  config.env.comparison = process.env.COMPARISON || false

  on('after:screenshot', details => {
    // We only want to modify the behavior of baseline and comparison runs.
    if (config.env.baseline || config.env.comparison) {
      // We keep track of the file name and number to make sure they are saved in the proper order and in their relevant folders.
      // An alternative would have been to look up the folder for the latest image, but this was the simpler approach.
      let lastScreenshotFile = ''
      let lastScreenshotNumber = 0

      // We append the proper suffix number to the image, create the folder, and move the file.
      const createDirAndRename = filePath => {
        if (lastScreenshotFile === filePath) {
        } else {
          lastScreenshotNumber = 0
        lastScreenshotFile = filePath
        const newPath = filePath.replace(
          ` #${lastScreenshotNumber}.png`

        return new Promise((resolve, reject) => {
          fs.mkdir(path.dirname(newPath), { recursive: true }, mkdirErr => {
            if (mkdirErr) {
              return reject(mkdirErr)
            fs.rename(details.path, newPath, renameErr => {
              if (renameErr) {
                return reject(renameErr)
              resolve({ path: newPath })

      const screenshotPath = `visualComparison/${config.env.baseline ? 'baseline' : 'comparison'}`

      return createDirAndRename(details.path
        .replace('cypress/integration', screenshotPath)
        .replace('All Specs', screenshotPath)
  return config

Then we invoked each of the runs by adding the corresponding environment variable to the Cypress call in the project’s package.json:

"scripts": {
  "cypress:baseline": "BASELINE=true yarn cypress:open",
  "cypress:comparison": "COMPARISON=true yarn cypress:open"

Once we ran our new commands, we could see that all the screenshots taken during the run were moved to the appropriate folders.


A screenshot showing images taken during the run and moved to folders.

Figure 3. Visual run results

Next, we tried to overwrite cy.get(), Cypress’s main command to return DOM elements, and take a screenshot of any elements called along with its default implementation. Unfortunately, cy.get() is a tricky command to change, as calling the original command in its own definition leads to an infinite loop. The suggested approach to work around this limitation is to create a separate custom command and then have that new command take a screenshot after finding the element:

Cypress.Commands.add("getAndScreenshot", (selector, options) => {
  // Note: You might need to tweak the command when getting multiple elements.
  return cy.get(selector).screenshot()

it("get overwrite", () => {

However, our calls to interact with elements on the page were already wrapped in an internal getElement() function. So all we had to do was make sure that a screenshot was taken when the wrapper was called.

Results Obtained Via Visual Regression Testing

Once we had the screenshots, the only thing left to do was merge them. For that, we created a simple node script using Canvas. In the end, the script enabled us to generate 618 comparison images! Some of the differences were easy to spot by opening the Talent Portal, but some of the issues weren’t as obvious.


Before and after example of incorrect usage of Picasso, showing red and black colors in the element.

Figure 4. Example of not following new Picasso guidelines; a difference was expected, but the new version should have had a red background and white text



Before and after example of a slightly broken component layout, showing misaligned text next to a checkbox in the "After" image.

Figure 5. Example of a slightly broken component layout

Adding Value to UI Testing

First of all, the added visual regression tests proved to be useful and uncovered a few issues we could have missed without them. Even though we were expecting differences in our components, knowing what was actually changed helped narrow down problematic cases. So, if your project has an interface but you aren’t yet performing these tests, get to it!

The second lesson here, and perhaps the more important one, is that we were once again reminded that perfect is the enemy of good. If we had ruled out the possibility of running visual regression tests for this release because there was no prior setup, we may have missed out on a few bugs during the migration. Instead, we agreed on a plan that, while not ideal, was fast to execute, we worked toward it, and it paid off.

For more details on implementing a robust visual regression pipeline in your project, please refer to Cypress’s visual testing page, select the tool that best fits your needs, and watch the tutorial videos.

Original article source at:

#cypress #testing 

What is GEEK

Buddha Community

Visual Regression Testing with Cypress

Software Testing 101: Regression Tests, Unit Tests, Integration Tests

Automation and segregation can help you build better software
If you write automated tests and deliver them to the customer, he can make sure the software is working properly. And, at the end of the day, he paid for it.

Ok. We can segregate or separate the tests according to some criteria. For example, “white box” tests are used to measure the internal quality of the software, in addition to the expected results. They are very useful to know the percentage of lines of code executed, the cyclomatic complexity and several other software metrics. Unit tests are white box tests.

#testing #software testing #regression tests #unit tests #integration tests

Mikel  Okuneva

Mikel Okuneva


Where To Learn Test Programming — July 2020 Edition

What do you do when you have lots of free time on your hands? Why not learn test programming strategies and approaches?

When you’re looking for places to learn test programming, Test Automation University has you covered. From API testing through visual validation, you can hone your skills and learn new approaches on TAU.

We introduced five new TAU courses from April through June, and each of them can help you expand your knowledge, learn a new approach, and improve your craft as a test automation engineer. They are:

These courses add to the other three courses we introduced in January through March 2020:

  • IntelliJ for Test Automation Engineers (3 hrs 41 min)
  • Cucumber with JavaScript (1 hr 22 min)
  • Python Programming (2 hrs)

Each of these courses can give you a new set of skills.

Let’s look at each in a little detail.

Mobile Automation With Appium in JavaScript

Orane Findley teaches Mobile Automation with Appium in JavaScript. Orane walks through all the basics of Appium, starting with what it is and where it runs.


“Appium is an open-source tool for automating native, web, and hybrid applications on different platforms.”

In the introduction, Orane describes the course parts:

  • Setup and Dependencies — installing Appium and setting up your first project
  • Working with elements by finding them, sending values, clicking, and submitting
  • Creating sessions, changing screen orientations, and taking screenshots
  • Timing, including TimeOuts and Implicit Waits
  • Collecting attributes and data from an element
  • Selecting and using element states
  • Reviewing everything to make it all make sense

The first chapter, broken into five parts, gets your system ready for the rest of the course. You’ll download and install a Java Developer Kit, a stable version of Node.js, Android Studio and Emulator (for a mobile device emulator), Visual Studio Code for an IDE, Appium Server, and a sample Appium Android Package Kit. If you get into trouble, you can use the Test Automation University Slack channel to get help from Orane. Each subchapter contains the links to get to the proper software. Finally, Orane has you customize your configuration for the course project.

Chapter 2 deals with elements and screen interactions for your app. You can find elements on the page, interact with those elements, and scroll the page to make other elements visible. Orane breaks the chapter into three distinct subchapters so you can become competent with each part of finding, scrolling, and interacting with the app. The quiz comes at the end of the third subchapter.

The remaining chapters each deal with specific bullets listed above: sessions and screen capture, timing, element attributes, and using element states. The final summary chapter ensures you have internalized the key takeaways from the course. Each of these chapters includes its quiz.

When you complete this course successfully, you will have both a certificate of completion and the code infrastructure available on your system to start testing mobile apps using Appium.

Selenium WebDriver With Python

Andrew Knight, who blogs as The Automation Panda, teaches the course on Selenium WebDriver with Python. As Andrew points out, Python has become a popular language for test automation. If you don’t know Python at all, he points you to Jess Ingrassellino’s great course, Python for Test Programming, also on Test Automation University.


In the first chapter, Andrew has you write your first test. Not in Python, but Gherkin. If you have never used Gherkin syntax, it helps you structure your tests in pseudocode that you can translate into any language of your choice. Andrew points out that it’s important to write your test steps before you write test code — and Gherkin makes this process straightforward.

first test case

The second chapter goes through setting up a pytest, the test framework Andrew uses. He assumes you already have Python 3.8 installed. Depending on your machine, you may need to do some work (Macs come with Python 2.7.16 installed, which is old and won’t work. Andrew also goes through the pip package manager to install pipenv. He gives you a GitHub link to his test code for the project. And, finally, he creates a test using the Gherkin codes as comments to show you how a test runs in pytest.

In the third chapter, you set up Selenium Webdriver to work with specific browsers, then create your test fixture in the pytest. Andrew reminds you to download the appropriate browser driver for the browser you want to test — for example, chromedriver to drive Chrome and geckodriver to drive Firefox. Once you use pipenv to install Selenium, you begin your test fixture. One thing to remember is to call an explicit quit for your webdriver after a test.

Chapter 4 goes through page objects, and how you abstract page object details to simplify your test structure. Chapter 5 goes through element locator structures and how to use these in Python. And, in Chapter 6, Andrew goes through some common webdriver calls and how to use them in your tests. These first six chapters cover the basics of testing with Python and Selenium.

Now that you have the basics down, the final three chapters review some advanced ideas: testing with multiple browsers, handling race conditions, and running your tests in parallel. This course gives you specific skills around Python and Selenium on top of what you can get from the Python for Test Programming course.

#tutorial #performance #testing #automation #test automation #automated testing #visual testing #visual testing best practices #testing tutorial

Tamia  Walter

Tamia Walter


Testing Microservices Applications

The shift towards microservices and modular applications makes testing more important and more challenging at the same time. You have to make sure that the microservices running in containers perform well and as intended, but you can no longer rely on conventional testing strategies to get the job done.

This is where new testing approaches are needed. Testing your microservices applications require the right approach, a suitable set of tools, and immense attention to details. This article will guide you through the process of testing your microservices and talk about the challenges you will have to overcome along the way. Let’s get started, shall we?

A Brave New World

Traditionally, testing a monolith application meant configuring a test environment and setting up all of the application components in a way that matched the production environment. It took time to set up the testing environment, and there were a lot of complexities around the process.

Testing also requires the application to run in full. It is not possible to test monolith apps on a per-component basis, mainly because there is usually a base code that ties everything together, and the app is designed to run as a complete app to work properly.

Microservices running in containers offer one particular advantage: universal compatibility. You don’t have to match the testing environment with the deployment architecture exactly, and you can get away with testing individual components rather than the full app in some situations.

Of course, you will have to embrace the new cloud-native approach across the pipeline. Rather than creating critical dependencies between microservices, you need to treat each one as a semi-independent module.

The only monolith or centralized portion of the application is the database, but this too is an easy challenge to overcome. As long as you have a persistent database running on your test environment, you can perform tests at any time.

Keep in mind that there are additional things to focus on when testing microservices.

  • Microservices rely on network communications to talk to each other, so network reliability and requirements must be part of the testing.
  • Automation and infrastructure elements are now added as codes, and you have to make sure that they also run properly when microservices are pushed through the pipeline
  • While containerization is universal, you still have to pay attention to specific dependencies and create a testing strategy that allows for those dependencies to be included

Test containers are the method of choice for many developers. Unlike monolith apps, which lets you use stubs and mocks for testing, microservices need to be tested in test containers. Many CI/CD pipelines actually integrate production microservices as part of the testing process.

Contract Testing as an Approach

As mentioned before, there are many ways to test microservices effectively, but the one approach that developers now use reliably is contract testing. Loosely coupled microservices can be tested in an effective and efficient way using contract testing, mainly because this testing approach focuses on contracts; in other words, it focuses on how components or microservices communicate with each other.

Syntax and semantics construct how components communicate with each other. By defining syntax and semantics in a standardized way and testing microservices based on their ability to generate the right message formats and meet behavioral expectations, you can rest assured knowing that the microservices will behave as intended when deployed.

Ways to Test Microservices

It is easy to fall into the trap of making testing microservices complicated, but there are ways to avoid this problem. Testing microservices doesn’t have to be complicated at all when you have the right strategy in place.

There are several ways to test microservices too, including:

  • Unit testing: Which allows developers to test microservices in a granular way. It doesn’t limit testing to individual microservices, but rather allows developers to take a more granular approach such as testing individual features or runtimes.
  • Integration testing: Which handles the testing of microservices in an interactive way. Microservices still need to work with each other when they are deployed, and integration testing is a key process in making sure that they do.
  • End-to-end testing: Which⁠—as the name suggests⁠—tests microservices as a complete app. This type of testing enables the testing of features, UI, communications, and other components that construct the app.

What’s important to note is the fact that these testing approaches allow for asynchronous testing. After all, asynchronous development is what makes developing microservices very appealing in the first place. By allowing for asynchronous testing, you can also make sure that components or microservices can be updated independently to one another.

#blog #microservices #testing #caylent #contract testing #end-to-end testing #hoverfly #integration testing #microservices #microservices architecture #pact #testing #unit testing #vagrant #vcr

Dejah  Reinger

Dejah Reinger


How to Do API Testing?

Nowadays API testing is an integral part of testing. There are a lot of tools like postman, insomnia, etc. There are many articles that ask what is API, What is API testing, but the problem is How to do API testing? What I need to validate.

Note: In this article, I am going to use postman assertions for all the examples since it is the most popular tool. But this article is not intended only for the postman tool.

Let’s directly jump to the topic.

Let’s consider you have an API endpoint example{{username}} when you send the get request to that URL it returns the JSON response.

My API endpoint is{{username}}

The response is in JSON format like below


  "jobTitle": "string",
  "userid": "string",
  "phoneNumber": "string",
  "password": "string",
  "email": "",
  "firstName": "string",
  "lastName": "string",
  "userName": "string",
  "country": "string",
  "region": "string",
  "city": "string",
  "department": "string",
  "userType": 0

In the JSON we can see there are properties and associated values.

Now, For example, if we need details of the user with the username ‘ganeshhegde’ we need to send a **GET **request to ** **

Now there are two scenarios.

1. Valid Usecase: User is available in the database and it returns user details with status code 200

2. Invalid Usecase: User is Unavailable/Invalid user in this case it returns status with code 404 with not found message.

#tutorial #performance #api #test automation #api testing #testing and qa #application programming interface #testing as a service #testing tutorial #api test

Aurelie  Block

Aurelie Block


Top 10 Automation Testing Tools: 2020 Edition

The demand for delivering quality software faster — or “Quality at Speed” — requires organizations to search for solutions in Agile, continuous integration (CI), and DevOps methodologies. Test automation is an essential part of these aspects. The latest World Quality Report 2018–2019 suggests that test automation is the biggest bottleneck to deliver “Quality at Speed,” as it is an enabler of successful Agile and DevOps adoption.

Test automation cannot be realized without good tools; as they determine how automation is performed and whether the benefits of automation can be delivered. Test automation tools is a crucial component in the DevOps toolchain. The current test automation trends have increased in applying artificial intelligence and machine learning (AI/ML) to offer advanced capabilities for test optimization, intelligent test generation, execution, and reporting. It will be worthwhile to understand which tools are best poised to take advantage of these trends.****

#automation-testing #automation-testing-tools #testing #testing-tools #selenium #open-source #test-automation #automated-testing