Differences in executable line detection between CI jobs #841
Description
I have noticed something odd when looking at the coverage lcov.info files from the individual jobs in https://github.com/trixi-framework/Trixi.jl/runs/3512265649: For example, I compared the files lcov-tree_part1-ubuntu-latest-1.6-x64.info and lcov-p4est_part1-ubuntu-latest-1.6-x64.info for the file src/equations/compressible_euler_equations_1d.jl, for the lines 259ff (the flux function).
The flux function is defined from lines 259 to 268 (inclusive), where the first line and the last lines are the function flux(...) signature and the end statement, respectively. In the first LCOV file lcov-tree_part1-ubuntu-latest-1.6-x64.info , we get
...
DA:254,100
DA:260,12680238
DA:261,12680238
DA:262,12680238
DA:264,12680238
DA:265,12680238
DA:266,12680238
DA:267,12680238
DA:288,67168
...
where DA means "line with executable code" and the numbers are "line number, execution count". That is, both the function flux(...) line (line 259) and the end line (line 268) are not counted as "executable", and are thus not instrumented (see https://manpages.debian.org/stretch/lcov/geninfo.1.en.html#FILES or https://twiki.cern.ch/twiki/bin/view/SPI/PrincipleOfGCOV for more details on DA and the LCOV file structure), while the function body was executed 12,680,238 times.
The second LCOV file, lcov-p4est_part1-ubuntu-latest-1.6-x64.info, is for P4est.jl tests, thus the 1D Euler equations will never be used. As expected, the respective line execution counts in the LCOV file are all zero:
DA:254,0
DA:259,0
DA:260,0
DA:261,0
DA:262,0
DA:264,0
DA:265,0
DA:266,0
DA:267,0
DA:286,0
However, what's clearly different is that now lines 259 and 267 are also marked as executable, although with count zero (as they should be).
My strong suspicion now is that this is what causes the weird (and wrong) effect in the final Coverage report:
Somehow, while merging the two reports in Coverage.jl, the zero-count from the P4est tests for line 259 is registered as "hey, this line is executable but wasn't", while the Tree tests do not consider this line as executable and thus do not report any execution count at all. The end statement is obviously successfully discarded, but the function statement isn't - which causes the final report to contain a zero-count for the function statement.
Finally, the culprit seems to lie either with the parallel Coverage.jl execution, which extracts the line counts from the gcov files and does a bad job in filtering non-executable lines. Alternatively, the error is in the merging routine which should have discarded the line that is not reported at all by one of the measurements as executable.
Now, what to do with this? One possibility is to try to comb through the Coverage.jl code; alternatively we could open an issue there and ask about this behavior. Any suggestions?