Import coverage generation module from Zephyr

[nmeum]

Contribution description

This PR imports a slightly modified version of the coverage generation code from Zephyr and allows generating coverage information on real hardware (not just native) for GNU Gcov. The imported sys/coverage module writes coverage information on exit() to the UART. The output generated by sys/coverage can then be parsed using the dist/tools/coverage/coverage.py script which generates .gcda files from the output. These files can then be passed to Gcov frontends like gcovr or lcov to generate HTML coverage visualizations.

See also:

• https://docs.zephyrproject.org/1.14.1/guides/coverage.html
• https://github.com/zephyrproject-rtos/zephyr/tree/main/subsys/testsuite/coverage
• https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/gen_gcov_files.py

Testing procedure

A simple test can be performed with a hello-world application modified as follows:

diff --git a/examples/hello-world/Makefile b/examples/hello-world/Makefile
index 258d8e9baf..944f944d22 100644
--- a/examples/hello-world/Makefile
+++ b/examples/hello-world/Makefile
@@ -15,4 +15,6 @@ DEVELHELP ?= 1
 # Change this to 0 show compiler invocation lines by default:
 QUIET ?= 1
 
+USEMODULE += coverage
+
 include $(RIOTBASE)/Makefile.include
diff --git a/examples/hello-world/main.c b/examples/hello-world/main.c
index f51bf8c0a0..77c19d2c7e 100644
--- a/examples/hello-world/main.c
+++ b/examples/hello-world/main.c
@@ -20,6 +20,7 @@
  */
 
 #include <stdio.h>
+#include <stdlib.h>
 
 int main(void)
 {
@@ -28,5 +29,6 @@ int main(void)
     printf("You are running RIOT on a(n) %s board.\n", RIOT_BOARD);
     printf("This board features a(n) %s MCU.\n", RIOT_MCU);
 
+    exit(0);
     return 0;
 }

Compile the application as follows:

$ BOARD=nucleo-f401re make -C examples/hello-world/

Flash the application using:

$ BOARD=nucleo-f401re make -C examples/hello-world/ flash

Afterwards wait a few seconds and run make term as follows:

$ BOARD=nucleo-f401re make -C examples/hello-world/ term | tee /tmp/log.txt

Reset the board and wait until GCOV_COVERAGE_DUMP_END appears in the output, close the terminal as soon as it does and run the following command:

$ ./dist/tools/coverage/coverage.py -i /tmp/log.txt

If you see errors about invalid UTF-8 sequences, check the log.txt and remove these manually for now and re-run the script (see To-Do below). If the script terminates successfully, this should cause various .gcda files to be generated (find . -name '*.gcda'). These files can then be passed to a Gcov frontend (e.g. gcovr) to generate HTML files:

$ gcovr -r . --html -o coverage.html --html-details

Finally open coverage.html in a web browser.

To-Do

I think the tooling could be improved significantly. For example, a make coverage target which works similar to make term but invokes the ./dist/tools/coverage/coverage.py script directly and terminates as soon as it sees GCOV_COVERAGE_DUMP_END would be nice to have IMHO. Furthermore, I occasionally noticed UTF-8 decode errors when using the python script since resetting the board sometimes causes invalid UTF-8 sequences to be generated. This should probably be addressed as well. In general, I haven't tested this a lot yet and would welcome general feedback on the idea and the integration.

[miri64]

Would it make sense to provide a make coverage target, that

1. does everything (or most of) what you describe in the Testing procedure section
2. downloads the sources as a package from the Zephyr repo
   ?

BTW, is it possible to combine the results of the coverage tests? If so we could integrate that into our CI in the future, maybe.

[miri64]

For example, a make coverage target which works similar to make term but invokes the ./dist/tools/coverage/coverage.py script directly and terminates as soon as it sees GCOV_COVERAGE_DUMP_END would be nice to have IMHO.

I guess that would then be make coverage-term with my proposal.

[nmeum]

It should theoretically be possible to ship the coverage generation code from Zephyr in ./pkg instead of ./sys with custom RIOT patches on top. However, I am not sure what one would use as a $PKG_URL then. Cloning the entire zephyr repository for two files seems a bit overkill. I am not sure if there is any better way to checkout the source with the RIOT package system.

BTW, is it possible to combine the results of the coverage tests? If so we could integrate that into our CI in the future, maybe.

What do you mean by combine?

[miri64]

BTW, is it possible to combine the results of the coverage tests? If so we could integrate that into our CI in the future, maybe.

What do you mean by combine?

E.g. if I run examples/gnrc_border_router and fuzzing/gnrc_tcp, will the results be combined when piping to /tmp/log.txt or will the coverage.py script be just confused in that case. I am asking, since we most likely will never have a one-fits-all application. tests/unittests used to be a try on that, but as we know, that has its own (size) problems. And this does not even consider different execution paths due to module selection and #ifdefs.

[nmeum]

Ah, I think there are some Gcov frontends which allow merging coverage data. For example lcov allows merging multiple trace files by passing --add-tracefile multiple times. As such, it should be possible to run examples/gnrc_border_router once, generate lcov traces for it from the .gcda files and afterwards run fuzzing/gnrc_tcp generate trace files for this application as well and finally merge the two traces.

[miri64]

It should theoretically be possible to ship the coverage generation code from Zephyr in ./pkg instead of ./sys with custom RIOT patches on top. However, I am not sure what one would use as a $PKG_URL then. Cloning the entire zephyr repository for two files seems a bit overkill. I am not sure if there is any better way to checkout the source with the RIOT package system.

Indeed, but given that Zephyr did not put it in its own repository, that seems indeed to be overkill... How long does the fetch take with --depth=1 roughly? Maybe that's enough. What do you think @fjmolinas and @jia200x?

[kaspar030]

Indeed, but given that Zephyr did not put it in its own repository, that seems indeed to be overkill... How long does the fetch take with --depth=1 roughly? Maybe that's enough. What do you think @fjmolinas and @jia200x?

just copy those files over

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[miri64]

Ping @nmeum?

[miri64]

Ping? I ran into a situation where I would be very interested in that :-)

[nmeum]

Ping? I ran into a situation where I would be very interested in that :-)

Sorry, I didn't have the time to work further on this so far. However, last time I used this it did actually work fine. Is there anything that I need to do apart from rebasing and resolving merge conflicts? :)

[miri64]

Sorry, I didn't have the time to work further on this so far. However, last time I used this it did actually work fine. Is there anything that I need to do apart from rebasing and resolving merge conflicts? :)

From #17000 (comment)

It should theoretically be possible to ship the coverage generation code from Zephyr in ./pkg instead of ./sys with custom RIOT patches on top. However, I am not sure what one would use as a $PKG_URL then. Cloning the entire zephyr repository for two files seems a bit overkill. I am not sure if there is any better way to checkout the source with the RIOT package system.

and then the conclusion #17000 (comment)

Indeed, but given that Zephyr did not put it in its own repository, that seems indeed to be overkill... How long does the fetch take with --depth=1 roughly? Maybe that's enough. What do you think @fjmolinas and @jia200x?

just copy those files over

Not sure if this was addressed. Other than that, I guess the dedicated make target and the merging of coverage data from several tests can be addressed in a separate PR, once this is in place.

[miri64]

Oh, now after looking at the code again, maybe, instead of adding the dumping to the exit() function, it would make sense to add it to the main_trampoline() in core/lib/init.c (after the call of main()) instead. This way, the code of an application does not need to be touched to include the coverage dump.

[nmeum]

• Rebased against current master
• Fixed a few bugs
• Improved the documentation and moved it to Doxygen

I also tested the current version again using BOARD=nucleo-f401re.

With this board, I was able to successfully generate coverage for examples/hello_world with gcovr.

@miri64 could you take another look and maybe test if it also works for you? :)

[nmeum]

Not sure if this is the best subsection for the coverage documentation.

Maybe it makes sense to rename this to Developer Tooling or something along those lines?

[benpicco]

I like how little changes this needs in the code base.
Just some CI nits and a rebase is needed, I think this is then good to merge.

[nmeum]

Applied @benpicco's suggestions (thanks!) and rebased against master.

[benpicco]

CI still has come complaints about formatting.

[benpicco]

Please squash

[riot-ci]

Murdock results

❌ FAILED

ce479d8 tests: add test application for coverage module

Success	Failures	Total	Runtime
5198	0	6809	07m:44s

Artifacts

[benpicco]

Please add a test application that makes use of the coverage module so this gets build by CI.
I already fail to compile the example on native.

If the call to exit(0) is mandatory, would it make sense to use test_utils_main_exit_cb() for this?

It would only work if the test is run in the main thread, but I think that's the case for the kind of coverage reports we could create here.

[nmeum]

Please add a test application that makes use of the coverage module so this gets build by CI.

Good idea, will do so later.

If the call to exit(0) is mandatory, would it make sense to use test_utils_main_exit_cb() for this?

No, it's no longer mandatory. I haven't updated the original PR description.

See the Doxygen Documentation and #17000 (comment)

[nmeum]

Added a preliminary coverage test application which checks that a coverage dump is properly performed. Ideally, it would also be cool to check the validity of the obtained data by generating gcov files via dist/tools/coverage/coverage.py and processing them further using a gcov frontend but that will require additional software etc. so maybe we can add that later.

[benpicco]

When I run tests/coverage on native it immediately segfaults. 😕

make coverage still requires the board to be reset by the user.

Could this be solved by adding the reset target as a dependency and including the test_utils_interactive_sync module so output only starts when the host script gives the start command?

[nmeum]

When I run tests/coverage on native it immediately segfaults.

This is not intended to be run on native, on native you can directly use gcov instrumentation to determine coverage. Still interesting that it segfaults though, what does the backtrace look like?

make coverage still requires the board to be reset by the user.

Could this be solved by adding the reset target as a dependency and including the test_utils_interactive_sync module so output only starts when the host script gives the start command?

Sounds promising, I will look into it.

[miri64]

FYI (since AFAIK @nmeum is not on the forum): https://forum.riot-os.org/t/unit-test-code-coverage-using-gcov/2387/7