A solid testing strategy is a key point to the successful adoption of agile development. Test coverage is a metric used to measure how much of the source code of a program is executed by running a set of tests. It helps developers to identify the code in their application that was not tested.

Ideally, tests against software should define all behaviors of the software. However, this is rarely realized. That is how test coverage comes into play.

Test coverage is a visual measurement used to express which lines of code were executed by a test suite. This helps the software developers to locate where they should write new test cases to cover their source code completely.

In this post, we will introduce to you how Nebula Graph, a distributed graph database, performs test coverage.

Collect Information for Test Coverage

Nebula Graph is mainly developed in C++. And it supports most Linux environments and gcc/clang compilers. Thus through the support provided by these tool chains, we can easily perform coverage analysis in Nebula Graph.

Both gcc and clang support gcov-style test coverage, which is also very simple to use. The main steps are as follows:

1. Add compilation option --coverage -O0 -g
2. Add link option --coverage
3. Run the test
4. Integrate the test coverage report with lcov. For example lcov --capture --directory . --output-file coverage.info
5. Filter out coverage data for files that we aren’t interested in. For example lcov --remove coverage.info '*/opt/vesoft/*' -o 

By now the test coverage information has been successfully collected.

Next, you can generate a report through a tool such as genhtml and view the test coverage through your browser. See an example below:

But generating reports manually is very tedious. It is not automatic, and you have to perform such operations manually every time in the continuous development process. So the common practice is to set up code coverage for tests in CI and integrate with the third-party platforms (such as codecov, coveralls).

By doing so, developers do not have to care about the collection and report of test coverage, they only need to check the test coverage on the third-party platforms after committing their code.

What’s more, some third-party tools support integrating the test report with pull requests, which is very convenient.

Integrate With CI GitHub Action

There are many mainstream CI platforms, such as Travis, azure-pipelines and GitHub Action. Nebula Graph selects GitHub Action. In our previous post Automating Your Project Processes with Github Actions, we have introduced GitHub Action in detail, you may take a look at it to gain some basic understanding about Action.

Compared with other CI platforms, GitHub Action integrates better with GitHub and supports many operating systems and CPUs.

Furthermore, Action is powerful, simple and easy to use. The following example config a test coverage job in the workflow file:

Shell

1

- name: CMake with Coverage

2

  if: matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'

3

  run: |

4

    cmake -DENABLE_COVERAGE=ON -B build/

We can see that here we manage the coverage-related compilation options introduced above through a cmake option, which easily enables and disables the collection of coverage information.

For example, developers usually turn off coverage during development process, compilation, and testing to avoid the extra cost of compiling and running tests.

Shell

1

- name: Testing Coverage Report

2

  working-directory: build

3

  if: success() && matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'

4

  run: |

5

    set -e

6

    /usr/local/bin/lcov --version

7

    /usr/local/bin/lcov --capture --gcov-tool $GCOV --directory . --output-file coverage.info

8

    /usr/local/bin/lcov --remove coverage.info '*/opt/vesoft/*' -o clean.info

9

    bash <(curl -s https://codecov.io/bash) -Z -f clean.info

The piece of code here collects, merges and uploads the test reports to the third-party platform. This has been described in detail in the previous section. The operation of CI is shown in the following figure:

#tools #database #tutorial #software testing #graph database #ci cd #test coverage