Vanilla repository code coverage reporting

Unknown
Unknown
in Dev & Ops

As of today, Dec 4th, our continuous integration on the vanilla/vanilla now includes reporting on the code coverage of our PHP code.

Comments

The bot will collect code coverage from your pull request and make a comment that looks like this

image.png

It will report on the how much the coverage has changed from your target branch, and how much of the code that you touched in your PR is covered.

In this example:

  • Total code coverage has increased compared to master.
  • 100% of lines of code changed in the PR are covered by tests.

Checks

These are also incorporated into the status checks for a PR.

image.png

In this case, the PR in question

  • Project - Reduces the overall coverage of the project (Previously was 49.81% and is now 49.78%).
  • Patch - Covers on average less lines of its own than in the existing project. In order to pass this the PR must cover a minimum of 49.81% of its own lines to pass.

What to do if your PR is failing due to coverage issues?

Add more tests!

Alternatively if your PR is in such a rush, or is urgent enough that we do not have time to cover it with tests, 2 reviewers may still choose to merge over the PR, even without sufficient coverage.

Feedback

If you have any feedback about the coverage requirements implemented here, please discuss below. This will likely have some effect on our velocity in the near-term, but that is the intention. To illuminate gaps in our test coverage, and force us to slow down a little and make sure our pull requests are always an improvement to the project.

Comments

  • Unknown
    Unknown

    Just to followup here, it's been reporting for a while now and our coverage is almost up to 52%!

    One thing to note, if you are see a surprising coverage number on your pull request, one of the following things is likely:

    • Your previous pull request build failed one of the suites of unit tests. This causes all coverage from that suite to be ignored.
    • You need to merge master into your branch to see an accurate number.

    This issue probably has a more accurate description than I could come up with here:

    https://github.com/codecov/codecov-bash/issues/83

Categories