Automated "test_all" target and Code Coverage

[program--]

This PR reworks the test_all target to automatically amalgamate all tests defined through ngen_add_test, and builds on #348 to add code coverage using gcovr instead of lcov. In addition to using gcovr, modifications were made to CodeCoverage.cmake to support using llvm-cov with gcov emulation.

Additions

• CMake option NGEN_WITH_COVERAGE. Dependent on NGEN_WITH_TESTS, and allows building the ngen_coverage_* targets.
• cmake/CodeCoverage.cmake module, which provides functions for generating HTML and XML code coverage documents.
• Targets ngen_coverage_html and ngen_coverage_xml for HTML and XML documents respectively.
  The output files for these targets are outputted to:
  • <build_directory>/<target_name>/index.html
  • <build_directory>/<target_name>.xml.

Changes

• Target test_all is now automatically generated through ngen_add_test rather than explicitly at the end of the test/CMakeLists.txt file.

Notes

• When generating code coverage documents, the .gcda files are generated alongside the source object files in CMakeFiles/ directories. As a result, an out-of-source build is necessary, since it is not worth attempting to clean up these files in an in-source build. Moreover, to generate successive coverage documents, the gcov data files need to be deleted beforehand, so deleting the out-of-source build directory is the recommended approach.

• If using a custom compiler with (e.g.) -DCMAKE_CXX_COMPILER=/path/to/bin/gcc-23, the path to the corresponding coverage analysis tool needs to be set with -DGCOV_PATH=/path/to/bin/gcov-23, or -DLLVM_COV_PATH=/directory/with/llvm/bin/llvm-cov-42 as appropriate

• Example Coverage:

  

Testing

• HTML code coverage generated using GCC, Clang, and IntelLLVM.

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Target Environment support

• Linux
• MacOS

[PhilMiller]

Thanks for doing this. It all looks good to me. I'll give it a quick run on macOS, too

[PhilMiller]

Would it be possible/unreasonable to enable coverage analysis in the CI pipeline builds?

[PhilMiller]

I just tested successfully on macOS with compilers from Homebrew. In addition to setting -DCMAKE_C_COMPILER=$(which gcc-13) -DCMAKE_CXX_COMPILER=$(which g++-13) -DCMAKE_Fortran_COMPILER=$(which gfortran-13), I had to set -DGCOV_PATH=$(which gcov-13) to match. The result without that, of running the system's /usr/bin/gcov is that it will silently run and produce output with no files or functions listed.

[PhilMiller]

Once the gcov/llvm-cov selection logic is fixed, I'm enthusiastically +1 on this

[PhilMiller]

Something to potentially enhance down the line would be to exclude irrelevant bits from the metrics. The worst offender is the Google Test source code in test/googletest/googletest, since we don't care that we use very little of it.

If it's easy to do that, it would be great to fold that in here

[PhilMiller]

It may also make sense to not measure coverage of the test files as such, and only report on src/ and include/, but that's a mixed bag - if there's code in the tests that's not running, it should either be activated or deleted.

[PhilMiller]

I discovered in testing this on my Mac that the exclusions that are supposed to limit results to just src/ and include/ get confused if the paths involve a symlink. Building through absolute paths instead resolves that.

I'm trying to patch that by resolving the paths, but it's not working as desired.

[PhilMiller]

There's maybe some race condition if one runs make -j 8 ngen_coverage_html de-novo in a freshly-configured tree, where the data seem not to be fully populated in the output. I'm not sure I can be bothered to care, but again, noting it here to document "don't hold it that way"

[PhilMiller]

My remaining comments are probably fine to ignore. This achieves its purpose, and isn't otherwise making anything else around it worse. Let's just get it in, reap the benefits, and we can fix nitty-gritty issues as they become practical concerns.

[program--]

@PhilMiller I've modified the CMake module to include your suggestions, but particularly, removed all the language-specific flags and used add_compile_options/link_libraries as previously done. Should be good now 👍🏾