RFC: Deeper code coverage profiling with Twister test automation

[golowanow]

Introduction

Twister tool supports dynamic code coverage profiling on test execution; it also generates human readable reports in various formats.
Currently this Twister feature provides less data than it is possible to achieve effectively on a CI pipeline 'at scale' for precise code profiling, test planning and gap analysis.
The resulting code coverage report might be also misleading without its test scope context.

This proposal is to extend Twister in two aspects:

• compose code coverage reports at deeper level - per test instance in addition to the aggregated report for all tests in the running scope.
• convey coverage results as an integral part of its test suite execution result.

Problem description

• Twister currently process the GCOV coverage tool's output (.gcno, .gcda files) and aggregates coverage statistics from all the test configurations which it executes for the test plan requested, so the final coverage report shows the test plan coverage summary, without details on how each test instance configuration built contributes to the statistics (unless the test plan is just one test configuration); worse if the test plan is for several platforms.
• moreover, there is no easy way to identify each test suite instance's contribution in the final report, e.g. to compare and find subtle profiling differences of each build configuration.
• In case of a test failure (e.g. because of occasional stack overflow by additional coverage data), or a test passed, but its coverage data garbled/truncated, - these problems are hidden in the total numbers.
• The coverage data files are separate artifacts and an additional effort is needed to bind them to the proper test execution data.

Proposed change

1. Twister (optionally) to generate separate coverage reports for each test suite configuration it executes.
2. Twister (optionally) to include coverage data into twister.json as additional properties of its test suite run together with the test results.

Detailed RFC

1. The new --coverage-split mode extends Twister code coverage operations to report coverage statistics on each test instance configuration run individually in addition to the currently supported (default) reporting mode which aggregates data into a single
   report summarizing results of all the tests in the current scope of the Twister run.
   The split mode allows to identify precisely what amount of code coverage each test case configuration provides and to analyze its contribution to the overall test plan's coverage.
   Each build directory will have its own coverage report and data files, so the overall disk space and the total execution time increase.
   The split mode reads the raw data files while the test jobs are running in parallel, so the report aggregation stage doesn't need to do the same at its turn.
   Another new --disable-coverage-aggregation option allows to execute only the --coverage-split mode when the summary statistics are not needed.

2. New Twister option --coverage-sections allows to select what coverage data to include into the twister.json file for its further analysis in addition to the test meta-data and the test results.
   The choices are to select either the coverage summary, or the detailed branch coverage, ztest coverage, or all the coverage data collected.
   Also the coverage run 'status' and coverage 'tool' values are added with any non-default choice. The 'environment' top object has additional 'gcov_tool' property to keep the command line argument selection.
   Depending on --coverage-split and --disable-coverage-aggregation options the coverage data is attached either to its test suite object or/and at the report's top level as the aggregated summary of the test plan execution scope.

Proposed change (Detailed)

see #66345 in review with the proposed Twister extension.

Dependencies

The change should not affect any CI validation pipeline which uses Twister, unless the new reporting features are enabled explicitly.
With the new options enabled gradually (split aggregation, selected extra json sections) more data will be collected eventually increasing twister.json size and additional coverage reports.
The overall time for Twister to execute the same test plan on the same host(s) will also grow, although with better workload distribution - Twister will process GCOV output as parallel jobs instead of doing it once at the final report generation, as currently.

Concerns and Unresolved Questions

• The additional code coverage data in twister.json is fully supported only for the default gcovr reporting tool as it depends on gcovr capability. For lcov reporter the coverage run status is only added.
• The additional data will be an expected challenge to analyze, but isn't it a goal to have more data ?
• More complex json data. For example, the ElasticSearch needs to parse the extended multi-level twister.json sections; to visualize it with Kibana it needs custom Vega standart charts. More details on this with an example script to upload the new code coverage data.

Alternatives

• Use additional custom scripts or other tools to collect and aggregate GCOV artifacts differently after the Twister has done its part of the test plan execution.
• Split the test plan Twister execution into many small jobs for each test case configuration to execute it separately with its own code coverage reports.
• Don't use Twister for code coverage collection.

[golowanow]

@JordanYates, I've noticed you are working on twister + coverage improvements.
Could you also have a look at this RFC and related #66345 ?