C++ branch coverage false positives

[carun]

When using g++, there are many false positives with branch coverage, caused by standard operators and libraries, caused by if (ptr != NULL) checks employed by the compiler. Is there a way to ignore such false positives? False positives with printf is not persistent if we use a gcc.

Sample code

#include <cstdio>
#include <cstring>
#include <iostream>
#include <string>

int main(int argc, const char *argv[])
{
    bool b = false;
    if (strcmp(argv[1], "1") == 0)
        b = true;

    char *a = nullptr;
    if (b)
      printf("Hai\n");
    delete[] a;

    std::string str("asdads");
    str = "cd";

    std::cout << str << std::endl;
    return 0;
}

Steps

$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
$ g++ 1.cpp -fprofile-arcs -ftest-coverage -g -o 1 -std=c++11
$ lcov -c --rc lcov_branch_coverage=1 --no-external -d . -o 1.info
$ genhtml -s  --rc lcov_branch_coverage=1 --legend 1.info -o o

Output

With GCC 4.8.5 on RHEL7

[henry2cox]

Yes. See my earlier replies WRT ‘branch filtering’ – which is implemented in my version of the code. See “differential coverage” and the related descriptions and pull request.

[carun]

Reference PR of @henry2cox: #86

[carun]

@henry2cox your branch at https://github.com/henry2cox/lcov/tree/8728189f3e04a3efe0c2778de6c4a2e0cdb0d17d is unstable. I see incorrect result for the same input.

Reading data file 1.info
Found 8 entries.
Found common filename prefix "/usr/include/c++/9"
Writing .css and .png files.
Generating output.
processing dir /tmp/a
Processing file /tmp/a/1.cpp
  lines=12 hit=12
processing dir /usr/include/c++/9/bits
Processing file bits/stl_iterator_base_types.h
  lines=2 hit=0
Processing file bits/basic_string.h
  lines=6 hit=0
Processing file bits/char_traits.h
  lines=2 hit=0
Processing file bits/stl_iterator_base_funcs.h
  lines=5 hit=0
Processing file bits/basic_string.tcc
  lines=13 hit=0
processing dir /usr/include/c++/9/ext
Processing file ext/new_allocator.h
  lines=2 hit=0
Processing file ext/type_traits.h
  lines=2 hit=0
Writing directory view page.
Overall coverage rate:
  lines......: 27.3% (12 of 44 lines)
  functions......: 9.1% (1 of 11 functions)
  branches......: 23.8% (10 of 42 branches)

[henry2cox]

I see a different result than you do...works for me.
$ g++ -fprofile-arcs -ftest-coverage -g -o foo -std=c++11 foo.cpp
$ ./foo 1
$ lcov --rc lcov_branch_coverage=1 --no-external -d . -o foo.info -c
Capturing coverage data from .
geninfo cmd: '/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/bin/geninfo . --output-filename foo.info --no-external --rc lcov_branch_coverage=1'Found gcov version: 5.2.0
Using intermediate gcov format
Scanning . for .gcda files ...
Found 1 data files in .
Processing foo.gcda
Finished .info-file creation
$ ../../bin/genhtml -s --rc lcov_branch_coverage=1 --branch-coverage --legend foo.info -o rpt
Reading data file foo.info
Found 2 entries.
Found common filename prefix "/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests"
Writing .css and .png files.
Generating output.
processing dir /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0
Processing file /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0/iostream
lines=1 hit=1
processing dir /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo
Processing file foo/foo.cpp
lines=13 hit=13
Writing directory view page.
Overall coverage rate:
lines......: 100.0% (14 of 14 lines)
functions......: 100.0% (3 of 3 functions)
branches......: 50.0% (10 of 20 branches)

Running again with branch filtering...
$ genhtml -s --rc lcov_branch_coverage=1 --branch-coverage --filter branch --legend foo.info -o rpt2
Reading data file foo.info
reading /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo/foo.cpp (for bogus branch filtering)
Found 2 entries.
Found common filename prefix "/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests"
Writing .css and .png files.
Generating output.
processing dir /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0
Processing file /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0/iostream
lines=1 hit=1
processing dir /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo
Processing file foo/foo.cpp
lines=13 hit=13
Writing directory view page.
Overall coverage rate:
lines......: 100.0% (14 of 14 lines)
functions......: 100.0% (3 of 3 functions)
branches......: 50.0% (2 of 4 branches)
Filter suppressions 'branch':
6 instances
16 coverpoints

[carun]

Nice to see the results. So here is what I did on my Ubuntu 20.04.1 LTS. Can you tell me what I'm missing?

2020-11-13 12:06:26 ~/code/foss
$ git clone https://github.com/henry2cox/lcov
Cloning into 'lcov'...
remote: Enumerating objects: 142, done.
remote: Counting objects: 100% (142/142), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 2201 (delta 81), reused 110 (delta 63), pack-reused 2059
Receiving objects: 100% (2201/2201), 1003.74 KiB | 7.22 MiB/s, done.
Resolving deltas: 100% (1559/1559), done.

2020-11-13 12:06:31 ~/code/foss
$ cd lcov

2020-11-13 12:06:35 ~/code/foss/lcov (master)
$ git co -t origin/diffcov_initial
Branch 'diffcov_initial' set up to track remote branch 'diffcov_initial' from 'origin' by rebasing.
Switched to a new branch 'diffcov_initial'

2020-11-13 12:06:45 ~/code/foss/lcov (diffcov_initial)
$ sudo make install
bin/install.sh bin/lcov /usr/local/bin/lcov -m 755                                                                                                                                                          [5/28161]
bin/install.sh bin/genhtml /usr/local/bin/genhtml -m 755
(cd /usr/local/bin ; rm -f gendiffcov ; ln -s genhtml gendiffcov)
bin/install.sh bin/geninfo /usr/local/bin/geninfo -m 755
bin/install.sh bin/genpng /usr/local/bin/genpng -m 755
(cd /usr/local/bin ; rm -f gendiffpng ; ln -s genpng gendiffpng)
bin/install.sh bin/gendesc /usr/local/bin/gendesc -m 755
bin/install.sh bin/p4udiff /usr/local/share/lcov/support-scripts/p4udiff -m 755
bin/install.sh bin/p4annotate /usr/local/share/lcov/support-scripts/p4annotate -m 755
bin/install.sh bin/gitblame /usr/local/share/lcov/support-scripts/gitblame -m 755
bin/install.sh lib/lcovutil.pm /usr/local/lib/lcovutil.pm -m 755
bin/install.sh man/lcov.1 /usr/local/share/man/man1/lcov.1 -m 644
bin/install.sh man/genhtml.1 /usr/local/share/man/man1/genhtml.1 -m 644
(cd /usr/local/share/man/man1 ; rm -f gendiffcov.1 ; ln -s genhtml.1 gendiffcov.1)
bin/install.sh man/geninfo.1 /usr/local/share/man/man1/geninfo.1 -m 644
bin/install.sh man/genpng.1 /usr/local/share/man/man1/genpng.1 -m 644
(cd /usr/local/share/man/man1 ; rm -f gendiffpng.1 ; ln -s genpng.1 gendiffpng.1)
bin/install.sh man/gendesc.1 /usr/local/share/man/man1/gendesc.1 -m 644
bin/install.sh man/lcovrc.5 /usr/local/share/man/man5/lcovrc.5 -m 644
bin/install.sh lcovrc /usr/local/etc/lcovrc -m 644
bin/updateversion.pl /usr/local/bin/lcov 1.14 36.g8728189 1.14-36-g8728189
Updating bin tool /usr/local/bin/lcov
Done.
bin/updateversion.pl /usr/local/bin/genhtml 1.14 36.g8728189 1.14-36-g8728189
Updating bin tool /usr/local/bin/genhtml
Done.
bin/updateversion.pl /usr/local/bin/geninfo 1.14 36.g8728189 1.14-36-g8728189
Updating bin tool /usr/local/bin/geninfo
Done.
bin/updateversion.pl /usr/local/bin/genpng 1.14 36.g8728189 1.14-36-g8728189
Updating bin tool /usr/local/bin/genpng
Done.
bin/updateversion.pl /usr/local/bin/gendesc 1.14 36.g8728189 1.14-36-g8728189
Updating bin tool /usr/local/bin/gendesc
Done.
bin/updateversion.pl /usr/local/share/man/man1/lcov.1 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man1/lcov.1
Done.
bin/updateversion.pl /usr/local/share/man/man1/genhtml.1 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man1/genhtml.1
Done.
bin/updateversion.pl /usr/local/share/man/man1/geninfo.1 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man1/geninfo.1
Done.
bin/updateversion.pl /usr/local/share/man/man1/genpng.1 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man1/genpng.1
Done.
bin/updateversion.pl /usr/local/share/man/man1/gendesc.1 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man1/gendesc.1
Done.
bin/updateversion.pl /usr/local/share/man/man5/lcovrc.5 1.14 36.g8728189 1.14-36-g8728189
Updating man page /usr/local/share/man/man5/lcovrc.5
Done.
2020-11-13 12:07:08 ~/code/foss/lcov (diffcov_initial)
$
2020-11-13 12:08:11 ~/code/foss/lcov (diffcov_initial)
$ cd /tmp/a

2020-11-13 12:08:14 /tmp/a
$ lcov --version
lcov: LCOV version 1.14-36-g8728189

2020-11-13 12:09:36 /tmp/a
$ g++ 1.cpp -fprofile-arcs -ftest-coverage -g -o 1 -std=c++11

2020-11-13 12:09:38 /tmp/a
$ lcov -c --rc lcov_branch_coverage=1 --no-external -d . -o 1.info
Capturing coverage data from .
Found gcov version: 9.3.0
Using intermediate gcov format
Scanning . for .gcda files ...
Found 1 data files in .
Processing 1.gcda
Finished .info-file creation

2020-11-13 12:12:59 /tmp/a
$ genhtml -s  --rc lcov_branch_coverage=1 --legend 1.info -o o
Can't locate DateTime.pm in @INC (you may need to install the DateTime module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/x86_64-linux-gnu/perl5/5.3
0 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.30 /usr/share/perl/5.30 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/local/bin/genhtml line 86.
BEGIN failed--compilation aborted at /usr/local/bin/genhtml line 86.

2020-11-13 12:13:02 /tmp/a
$ sudo apt install -yqq libdatetime-perl

2020-11-13 12:13:26 /tmp/a
$ genhtml -s  --rc lcov_branch_coverage=1 --legend 1.info -o o
Can't locate DateTime.pm in @INC (you may need to install the DateTime module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/x86_64-linux-gnu/perl5/5.3
0 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.30 /usr/share/perl/5.30 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/local/bin/genhtml line 86.
BEGIN failed--compilation aborted at /usr/local/bin/genhtml line 86.

2020-11-13 12:13:53 /tmp/a
$ sudo apt install -yqq libdatetime-format-w3cdtf-perl

2020-11-13 12:13:58 /tmp/a
$ genhtml -s  --rc lcov_branch_coverage=1 --legend 1.info -o o
Reading data file 1.info
Found 8 entries.
Found common filename prefix "/usr/include/c++/9"
Writing .css and .png files.
Generating output.
processing dir /tmp/a
Processing file /tmp/a/1.cpp
  lines=12 hit=12
processing dir /usr/include/c++/9/bits
Processing file bits/stl_iterator_base_funcs.h
  lines=5 hit=0
Processing file bits/char_traits.h
  lines=2 hit=0
Processing file bits/basic_string.h
  lines=6 hit=0
Processing file bits/basic_string.tcc
  lines=13 hit=0
Processing file bits/stl_iterator_base_types.h
  lines=2 hit=0
processing dir /usr/include/c++/9/ext
Processing file ext/type_traits.h
  lines=2 hit=0
Processing file ext/new_allocator.h
  lines=2 hit=0
Writing directory view page.
Overall coverage rate:
  lines......: 27.3% (12 of 44 lines)
  functions......: 9.1% (1 of 11 functions)
  branches......: 23.8% (10 of 42 branches)

[carun]

Thanks for your help. Passing --filter branch option to lcov removes the branch false positives altogether. Good work.
However even with --no-external, genhtml produces coverage for external libraries. Let me dig a bit more.

[carun]

I've raised henry2cox#1 to fix the issue with --no-external. Please review.

[RafaelRey]

Are there any news about this topic?

[henry2cox]

Hi -
I think I will let Peter add any update with respect to schedule - but the fixes described above (now in PR #169) will be merged "soon". :-)
As discussed in other reports, please note that the filtering hack is something of a hack - and can be easily defeated by a bloody-minded user. You may also need to add 'noexcept' to some functions.
With respect to the 'no external' issue mentioned above: I can't reproduce the issue :-( (works for me).
Henry

[carun]

Resolved no-external issue by removing the option from ~/.lcovrc

[henry2cox]

I think this issue (bogus branch coverpoints) is addressed in SHA 5f659f6 (as discussed above).
I also completely agree that this approach is a workaround verging on ugly hack :-)

If you think that the issue is resolved: please close this issue.
If it isn't fixed: please clarify what else should be done. Note that all real solutions probably need to be done within the compiler infrastructure. I don't think that lcov will ever have such infrastructure (though tools feeding or filtering lcov might).

Henry

[carun]

Hey @henry2cox looks like your PR got merged. Congrats. Great work on that.

I'll test from the master branch of this repo when I get time and close this issue after that.

[carun]

Works as expected with --filter branch

[yi452873250]

I see a different result than you do...works for me. $ g++ -fprofile-arcs -ftest-coverage -g -o foo -std=c++11 foo.cpp $ ./foo 1 $ lcov --rc lcov_branch_coverage=1 --no-external -d . -o foo.info -c Capturing coverage data from . geninfo cmd: '/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/bin/geninfo . --output-filename foo.info --no-external --rc lcov_branch_coverage=1'Found gcov version: 5.2.0 Using intermediate gcov format Scanning . for .gcda files ... Found 1 data files in . Processing foo.gcda Finished .info-file creation $ ../../bin/genhtml -s --rc lcov_branch_coverage=1 --branch-coverage --legend foo.info -o rpt Reading data file foo.info Found 2 entries. Found common filename prefix "/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests" Writing .css and .png files. Generating output. processing dir /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0 Processing file /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0/iostream lines=1 hit=1 processing dir /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo Processing file foo/foo.cpp lines=13 hit=13 Writing directory view page. Overall coverage rate: lines......: 100.0% (14 of 14 lines) functions......: 100.0% (3 of 3 functions) branches......: 50.0% (10 of 20 branches)

Running again with branch filtering... $ genhtml -s --rc lcov_branch_coverage=1 --branch-coverage --filter branch --legend foo.info -o rpt2 Reading data file foo.info reading /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo/foo.cpp (for bogus branch filtering) Found 2 entries. Found common filename prefix "/home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests" Writing .css and .png files. Generating output. processing dir /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0 Processing file /mtkoss/gcc/5.2.0-rhel6/x86_64/include/c++/5.2.0/iostream lines=1 hit=1 processing dir /home/hcox/pwa/dspMethLibs/releng/coverage/lcov/tests/foo Processing file foo/foo.cpp lines=13 hit=13 Writing directory view page. Overall coverage rate: lines......: 100.0% (14 of 14 lines) functions......: 100.0% (3 of 3 functions) branches......: 50.0% (2 of 4 branches) Filter suppressions 'branch': 6 instances 16 coverpoints

ubuntu 2020.04 lcov 1.14
Follow your operation，but result is not same
could you tell me ，what`s wrong with me,please

[henry2cox]

wrong version.
looks like you are using lcov 1.14 (very, very old).
--filter option is much newer and is not yet available in any numbered release. You need to download latest version and install from github.

[carun]

Just curious, how does this get downstreamed to different distros, say the next version of Ubuntu? Do you notify them or do they track and pull the latest?

[henry2cox]

Now there you have me. No idea.
Perhaps Peter knows.

[yi452873250]

hi，I have updated the lcov ,bug not effect.it`s unblivable

[henry2cox]

Looks like you did not update to TOT (top of tree) - or you are not using the TOT version that you updated.
which genhtml might be instructive. genhtml --version might also be useful.
Similarly, just running /full/path/to/where/you/installed/genhtml ... would likely do what you wanted.

[khaiquangtran]

Hello @yi452873250
I think you should upgrade lcov 2.1 to use --filter branch
https://github.com/linux-test-project/lcov