Selenium Isn’t Flakey

Selenium Isn’t Flakey

How to improve pass/fail rate without re-writing your test suite. In 2017, the Vrbo™️ search page team (then called the HomeAway® search page team) rewrote the legacy search page application using modern JavaScript technologies such as React and Node. As part of this rewrite, we also wanted to transition from a manually QA-ed biweekly release cycle to a continuous deployment model with automated testing. Since we needed to support IE11, Selenium was the obvious choice for end-to-end release validation.

In 2017, the Vrbo™️ search page team (then called the HomeAway® search page team) rewrote the legacy search page application using modern JavaScript technologies such as React and Node. As part of this rewrite, we also wanted to transition from a manually QA-ed biweekly release cycle to a continuous deployment model with automated testing. Since we needed to support IE11, Selenium was the obvious choice for end-to-end release validation.

Since the rewrite, these Selenium tests have proven invaluable for catching page breaking issues (especially IE11) and have caught many P1 issues before making it out to production. Unfortunately, in the beginning of 2020 we noticed that our pass/fail rate for these tests started to approach 50%. This meant that our tests had gotten so flakey over time that developers spent double the amount of time re-running the suite to get a clean build. Something had to be done to get our pass rate back up to an acceptable level.

So begins the investigation…

During a team meeting we discussed what might be causing the flake. Was it our Selenium cloud provider, Saucelabs®? Was it our test environment? Was it Selenium itself? We went so far as to discuss whether it made sense to dump our test suite completely and start over with Puppeteer, but instead we opted to do a deep dive into the failing tests.

What did we find?

After spending almost a week exhaustively looking through Saucelabs screen-replays and logs we discovered several key issues leading to our test flake, which in order of flakiness included

1. Flakey testing environment

At Vrbo, we have 2 separate testing environments: test and stage. Traditionally test was reserved for developer testing whereas stage was reserved for manual pre-deploy testing and C-team demonstrations. Because of this distinction, test tended to not be stable and many services running in test were either under-provisioned or had poor test data. We found that almost half of all the Selenium test failures were caused by errant errors and timeouts in microservices in test that our app depended upon.

Because stage was better maintained due to its importance as a demo environment, we reconfigured out tests to run against an instance of our application running in stage rather than test. Since our tests didn’t rely on specific test data, this change was simple and led to a great reduction in errant errors and timeouts, but it still only reduced our failures by ~20% or so.

javascript selenium software-engineering programming technology

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Software Developer vs Software Engineer — Differences: Bogus or Real?

In this article, see if there are any differences between software developers and software engineers. What you’re about to read mostly revolves around my personal thoughts, deductions, and offbeat imagination. If you have different sentiments, add them in the comment section, and let’s dispute! So, today’s topic…

JavaScript Stranger Things

A collection of JavaScript oddities and quirks. JavaScript, bizarre and yet beautiful, has been a criticized programming language by the developer community, and they have some reasons why. The language was originally put together in ten days by Brendan Eich in 1995, then JavaScript has been standardized and heavily extended over the years and like most languages, it has its share of quirks and oddities.

Introduction to Deno: A Secure JavaScript & TypeScript Runtime

A JavaScript Runtime from the creator of NodeJs. Deno is a JavaScript and TypeScript runtime that lets you write programs in either languages and execute them from the command line. As for Deno, we can say that’s what makes Javascript run outside the browser, adding a series of features that it is not possible to find in the Javascript engine itself.

Selenium Online Training- Selenium Course with live Project| OnlineITGuru

Software company need expert with selenium certification. Learn best Java concepts in selenium training and become a testing engineer.

Interested in Learning to Program? 13 Reasons to Start Now

Software development is something that is gaining popularity at lightning speed with the development of technology. The demand for regular developers is high compared to most other mainstream professions. But, what are the other reasons for learning to code?