Automated testing is a crucial part of high-quality software development. There are several dos and don’ts and pros and cons with every level of automated testing.

If you are not familiar with the different levels of automation, you can go from testing a unit of code (ex. a class), to test the integration of these units (ex. multiple classes), to the whole system. Testing is all about the risks and picking the right compromises. Given this, you should attempt to have a fairly balanced test suite so you don’t end up with days of running automation and you can have a fast and reliable feedback loop.

In this post, I’ll be focusing on a testing framework that currently helps with testing the whole system from a web browser. These types of tests are usually called UI tests, end-to-end tests, or system tests. While system tests are great, as they exercise the system from a user’s perspective, there are a few issues with these.

• Reliability: These types of tests tend to fail a lot as you need the whole system up, and very brittle. Any minor hiccups, without the right level of retries, could cause a test failure.
• Expensive to troubleshoot: Without the proper logging and monitoring, these types of tests could be difficult to troubleshoot. Take as an example, a login page that doesn’t work. What is the exact issue? Could it be a frontend issue? a network issue? or, a backend issue?
• Duration: Because they require a UI and the backend to be working, there could be complex setup and long-running tasks. This could result in minutes per test in many cases.
• Maintainability: For these types of tests you tend to use CSS selectors. Picking up the wrong type of selectors might result in a failing test even though nothing is functionally wrong.

#javascript