Run coverage on tests

Table of Contents

I recommend running coverage on your tests.

Here’s a couple of reasons why, from the past couple of months.

Example one #

When writing tests, it’s common to copy and paste test functions, but sometimes you forget to rename the new one (see also: the Last Line Effect).

For example:

def test_get_install_to_run_with_platform(patched_installs):
    i = installs.get_install_to_run("<none>", None, "1.0-32")
    assert i["id"] == "PythonCore-1.0-32"
    assert i["executable"].match("python.exe")
    i = installs.get_install_to_run("<none>", None, "2.0-arm64")
    assert i["id"] == "PythonCore-2.0-arm64"
    assert i["executable"].match("python.exe")


def test_get_install_to_run_with_platform(patched_installs):
    i = installs.get_install_to_run("<none>", None, "1.0-32", windowed=True)
    assert i["id"] == "PythonCore-1.0-32"
    assert i["executable"].match("pythonw.exe")
    i = installs.get_install_to_run("<none>", None, "2.0-arm64", windowed=True)
    assert i["id"] == "PythonCore-2.0-arm64"
    assert i["executable"].match("pythonw.exe")

The tests pass, but the first one is never run because its name is redefined. This clearly shows up as a non-run test in the coverage report. In this case, we only need to rename one of them, and both are covered and pass.

But sometimes there’s a bug in the test which would cause it to fail, but we just don’t know because it’s not run.

Tip 1: This can also be found by Ruff’s F811 rule.

Tip 2: pytest’s parametrize is a great way to combine similar test functions with different input data.

Example two #

This is more subtle:

im = Image.new("RGB", (1, 1))
for colors in (("#f00",), ("#f00", "#0f0")):
    append_images = (Image.new("RGB", (1, 1), color) for color in colors)
    im_reloaded = roundtrip(im, save_all=True, append_images=append_images)

    assert_image_equal(im, im_reloaded)
    assert isinstance(im_reloaded, MpoImagePlugin.MpoImageFile)
    assert im_reloaded.mpinfo is not None
    assert im_reloaded.mpinfo[45056] == b"0100"

    for im_expected in append_images:
        im_reloaded.seek(im_reloaded.tell() + 1)
        assert_image_similar(im_reloaded, im_expected, 1)

It’s not so obvious when looking at the code, but Codecov highlights a problem:

The same code, but Codecov has flagged the last two lines were not covered

The append_images generator is being consumed inside roundtrip(), so we have nothing to iterate over in the for loop – hence no coverage. The fix is to use a list instead of a generator.

Header photo: Misplaced manhole cover (CC BY-NC-SA 2.0 Hugo van Kemenade).