Mutation testing in PHP project

Mutation testing in PHP project

Mutation testing. According to wikipedia, mutation testing involves applying small modifications to a software. These modifications are called mutations. Then, if you pass your tests and any of them fails because of this "mutation", then you have "killed the mutant"

Recently I spent an afternoon experimenting with mutation testing in PHP. In this post I would like to share the background, the main idea of mutation testing, and the lessons I’ve learned from it.

Mutation testing

In essence, mutation testing is a way of measuring the quality of a test suite. A tool generates a number of copies (mutants) of your source code under test, but modifies each of them in a small way. These small modifications are basically just errors that you as a programmer could have made, such as replacing a < with a <=. A high-quality testsuite would detect these modifications (kill the mutant) by failing one or more tests. In a suboptimal test suite it might happen that the tests remain green despite the modification to the source code. In such a case we speak about an escaped mutant. These present an opportunity to improve the test suite by adding a test that fails in the presence of the given modification. The Mutation Score Indicator (MSI, the percentage of mutants detected by the test set) provides a metric for the quality of the test suite.

My plan was to apply Infection to one of our internal libraries. I picked this library because of its high (line-based) code coverage, which would imply a high-quality test suite. Therefore I wondered what ‘leaks’ Infection could still find in it. The idea was to convert the escaped mutations found by Infection to new tests, working towards a PR to increase the MSI of the project as the main deliverable. This blog post wasn’t part of the original plan, but arose as an extra way of sharing some of the lessons learned along the way.

Lessons learned

Mutation testing is awesome!

My main takeaway was that mutation testing can really help you to improve your test suite. Even on a project with 96% line coverage, Infection found multiple scenarios that were not actually covered by the test suite.

One simplified example of this is the following. Suppose we have a function to generate a description for the amount of bottles of beer on the wall:

function describeBottles(int $amount = 99): string {
    return $amount . ' bottles of beer on the wall';

We could already have a test for this function like this:

class DescribeBottlesTest {
    public function testDescribesTheAmountOfBottlesOfBeer() {
        $this->assertSame('42 bottles of beer on the wall', describeBottles(42));

At first sight it may seem that this fully tests the function, and indeed the function shows up with all lines covered in a code coverage report. However, our tests do not check the default value for the argument. This means that some behavior of our function, i.e. that by default it describes 99 bottles, is not verified. Infection can uncover this when it produces a mutation like:

12) /tmp/bottles.php:2    [M] DecrementInteger

--- Original +++ New @@ @@ <?php

  • function describeBottles(int $amount = 99): string {
  • function describeBottles(int $amount = 98): string {
    return $amount . ' bottles of beer on the wall';

Here the DecrementInteger mutator has decremented an integer literal occurring in the source code, an error we could have made ourselves if we hit the wrong key on our keyboard. This would currently go unnoticed, but we can fix that by adding a test like:

class DescribeBottlesTest {
  public function testDescribes99BottlesByDefault() {
  $this-&gt;assertSame('99 bottles of beer on the wall', describeBottles());
} }

Other untested aspects commonly found by Infection were exception messages and some uncovered paths through complex logic. I added tests or assertions for most of these. For the logic, this is also a sign that the (cyclomatic/N-path) complexity is too high. Those pieces of code should be refactored, but I scheduled that for later.

On the first Infection run, without any changes to the test suite, it produced a 89% MSI. This is already quite good, but with some additions to the test suite I managed to raise the MSI to 93%.

‘False positives’

Getting the MSI much higher proved to be difficult though. Sometimes escaped mutants had changes to parts of the code that are nonessential details. In our view, these details do not constitute relevant behavior and are not part of the ‘contract’ of that unit. Why are they in the code then? Well, sometimes they have to be due to syntactical constraints. Take for example the following piece of code that may throw an exception:

// ...
try {
} catch (DuplicateDatabaseKeyException $e) {
  throw new UserAlreadyExistsExeption("User $username already exists", 0, $e); 

As explained in a previous blog post, we think it is important that exceptions are thrown at the right level of abstraction:

This requires catching and re-throwing exceptions like in the above code snippet. In that post, I also mentioned that we want to maintain the connection with the root cause by setting the $previous-parameter of the new exception. Due to PHP's syntax this requires one to also provide the $code-parameter, which we do not really use. We usually set it to 0 (the default), but honestly couldn't care less about its value.

Now the same DecrementInteger mutator could come along and produce this mutation:

53) /tmp/exception.php:289   [M] DecrementInteger

--- Original +++ New @@ @@ } catch (DuplicateDatabaseKeyException $e) {

  • throw new UserAlreadyExistsExeption("User $username already exists", 0, $e);
  • throw new UserAlreadyExistsExeption("User $username already exists", -1, $e); }

This mutation will not get caught by our test suite (because we do not assert the code of produced exceptions), so the mutant will escape. In this case we do not care however. We don’t expect our test suite to detect this, as (to us) the mutated code is just as fine as the original. We could add assertions for the exception code, but that would just be extra work without yielding extra value.

I think this is something to be aware of when doing mutation testing. Not all escaped mutants are necessarily bad. For each of them you have to ask yourself the question “Would it be bad if I made this ‘error’ in my code?”. If the answer is no, don’t bother about the escaped mutant.

What test to add?

One of the main difficulties when trying to kill a mutant was figuring out what kind of test to add. Just from seeing a changed line in the code it is not always clear how to write a test that would fail on the given line. This was further amplified by the fact that the codebase I worked with was written by a colleague, and I did not know it inside out yet.

I learned that the HTML code coverage report generated by PHPUnit can be of tremendous help here. If you hover over a covered line of code there, it shows you which tests cover that line. This way you can lookup which tests already exercise the mutated line of code. The test you want to add to kill the mutant is probably a variation of one of them. This reduces your problem to analyzing these ‘example’ tests and reasoning about what you could change in them to fail when the mutation is present.

It improves your code too

Not only the test suite got some updates during my experiments with mutation testing; the production code improved as well. Sometimes the mutants generated by Infection were actually better than the original version! One such case looked somewhat like this:

class SomeStore {
public static function createWithInMemoryDatabase(): self {
  $database = $database ?? new SqliteDatabase(':memory:');
  // ...

Among the generated mutations there was one that removed the $database ?? null coalesce operator. This is actually an improvement, as the null coalesce operator is useless here! At the start of the function, $database is always null, so the operator always resolves to its right-hand side, creating a new database. This code was an artifact from a moment when the method was named differently and allowed injecting a custom database through a $database parameter. Now that parameter has been removed, we can get rid of the null coalesce as well. While other static analysis tools could have found this dead code as well, at least Infection brought it to our attention.

Another example where the mutant turned out to be better was the removal of a trim() function. At that spot in the code there could never be any significant or problematic whitespace. The trim()-call thus was unnecessary and could be removed.


Once you’ve had some practice with mutation testing, it can really help with improving both your test suite and code. Infection is straightforward to setup and use, and makes it fairly simple to get started in PHP. Just keep in mind that not all escaped mutants are a problem and blindly striving for 100% MSI does not add value. Try it out, and let me know what your experiences are!

Thanks For Visiting, Keep Visiting. If you liked this post, share it with all of your programming buddies!

Further reading

Top 10 Testing Frameworks and Libraries for Java Developers

Unit vs E2E Testing for Vue.js

Testing Node.js with Mocha

React Native Web Testing

Testing Vue with Jest

Testing Flutter Apps - Making Sure Your Code Works

Testing your JavaScript Code — TDD

JavaScript Testing using Selenium WebDriver, Mocha and NodeJS

Originally published on

Angular 9 Tutorial: Learn to Build a CRUD Angular App Quickly

What's new in Bootstrap 5 and when Bootstrap 5 release date?

Brave, Chrome, Firefox, Opera or Edge: Which is Better and Faster?

How to Build Progressive Web Apps (PWA) using Angular 9

What is new features in Javascript ES2020 ECMAScript 2020

Learn Software Testing Course in Delhi - APTRON Solutions

Many institutes are having a Software Testing Training And Placement In Delhi but few of them are very great at teaching. In the event that you want to learn about software testing. We have designed this software testing training course to learn...

The best machine learning and deep learning libraries

You are asking Why TensorFlow, Spark MLlib, Scikit-learn, PyTorch, MXNet, and Keras shine for building and training machine learning and deep learning models.If you’re starting a new machine learning or deep learning project, you may be confused about which framework to choose...

Deep Learning vs. Conventional Machine Learning

Over the past few years, deep learning has given rise to a massive collection of ideas and techniques which are disruptive to conventional machine learning practices. However, are those ideas totally different from the traditional methods? Where are the connections and differences? What are the advantages and disadvantages? How practical are the deep learning methods for business applications? Chao will share her thoughts on those questions based on her readings and hands on experiments in the areas of text analytics (question answering system, sentiment analysis) and healthcare image classification.