What is the second number in BRDA? And why it is changed after merging tracefiles?

[myler]

I have two tracefile, one contains data collected on Linux, another contains data collected on Windows. After I used -a option to merge them into one overall tracefile, I found some BRDA's second number changed:
BRDA:7,0,2,0 changed to BRDA:7,1,2,0 if the Linux tracefile is behind Windows tracefile with -a option, vice versa.

[henry2cox]

man geninfo

[myler]

It seems no doc related to changing of block number in geninfo manpage, not sure the tool can recognize same branch of template instances built on Windows and Linux.

[myler]

https://github.com/linux-test-project/lcov/blob/master/lib/lcovutil.pm#L5230, this is what I searched, no chance to control the block number.
When we merge multiple tracefiles win_test.info, lin_test.info, ..., etc, we lose the information of the original branch, so that we cannot use BRDA data to track subsequent work, such as adding new tests to a uncovered branch.

[myler]

@henry2cox, I see you already added test https://github.com/linux-test-project/lcov/tree/master/tests/lcov/branch to cover this usage. For branch of template instances, the order of merging tracefiles definitely affects order of BRDAs. (The total number of BRDAs is not changed).

So, is there a sustainably way (e.g., unique id or mangled expr?) to track a branch is taken or not if we continuously merge new tracefiles?

[henry2cox]

If I understand the question, then I think I don't quite understand the issue you see.

Some background.
As you point out, the blockId is used to distinguish:

1. distinct conditionals which appear on the same line in the source code - e.g., foo = bar(a || b, c && d);, and
2. different conditionals which arise from the same statement - either:
   * in different compilation units (due to macro expansion) or
   * in different template instantiations (typically short-circuited Booleans involving template parameters)
   (unlike branch data, MC/DC data generated by gcc/14 doesn't have this issue).

The branch expression (if known) is used to decide who matches who.
It happens that certain languages (perl, Python, (System)Verilog) explicitly identify each branch expression (as a string - say a && !b) - but gcov-generated data for C/C++ does not. In gcov output, the branch expression is just an integer (related to expression traversal order in the compiler). This can be ambiguous when the same source line is used in multiple compilation units or when code changes over time.

lcov tries to sort this out in a couple of ways.

For the first case, it assumes that the order of gcov-generated output data for expressions on a particular line is the same in every result file. In this case, it also assumes (and checks) that the total number of distinct expressions (the number of distinct blockIds) and the number of branch records in each block is the same in each data file - and that the branch expressions (if known) are the same.

In the second case: it assumes that the related conditional is the same if the number of branch records and their branch expressions are the same.

Unfortunately: it isn't hard to see that there isn't sufficient information in the output data for lcov to distinguish the first and second cases.
Ambiguity is the result. :-(
Usually, this isn't too hard for a human to figure out - but it also depends on your infrastructure and your requirement - particularly if you are in a very high reliability environment.
In our environment, we find the coverage data to be actionable - but you may have a more complicated environment such that the tools are not sufficiently repeatable to create actionable results.

You mentioned that you are using windows - but you didn't mention your toolchain and toolchain version, or what the source of your input .info file data is (generated by gcc -> lcov(gcov) ? llvm -> llcov(llvm-cov)? llvm -> llvm-profdata -> llvm-cov? Something else?

Without seeing your actual data (or a representative small testcase), I am not sure what to suggest.
It is always possible that there is a bug in lcov data handling - and it isn't doing what it is supposed to be doing.
It is also possible that there are bugs in in gcov (or llvm-cov, or whatever extract tool you are using.)

Henry

[myler]

Thank you so much for this detail answer and so quick reply!!! This is very useful to me to understand how lcov works now.

but you didn't mention your toolchain and toolchain version

I‘m using Intel compiler (which is derived from llvm/clang), and the coverage data format is .gcno, and .gcda emitted by llvm-cov gcov.

Without seeing your actual data (or a representative small testcase)

Suppose we are comparing BRDAs in line 7 of example code:
Linux:

BRDA:7,0,0,- (line 23 never executed on Linux)
BRDA:7,0,1,- (line 23 never executed on Linux)
BRDA:7,0,2,0
BRDA:7,0,3,1

Windows:

BRDA:7,0,0,0 (line 16 only build on Windows)
BRDA:7,0,1,1 (line 16 only build on Windows)
BRDA:7,0,2,0
BRDA:7,0,3,1
BRDA:7,0,4,- (line 25 never executed on Windows)
BRDA:7,0,5,- (line 25 never executed on Windows)

Merged order linux info, then windows info: (lcov -a linux.info -a windows.info -o merged.info)

BRDA:7,0,0,-
BRDA:7,0,1,-
BRDA:7,0,2,0
BRDA:7,0,3,1
BRDA:7,1,0,0 (the <block number> changed from 0 to 1 after merging two info files if windows info file is after linux info file)
BRDA:7,1,1,1 (same as above)
BRDA:7,1,2,0 (same as above)
BRDA:7,1,3,1 (same as above)
BRDA:7,1,4,- (same as above)
BRDA:7,1,5,- (same as above)

Merged order windows info, then linux info: (lcov -a windows.info -a linux.info -o merged.info)

BRDA:7,0,0,0
BRDA:7,0,1,1
BRDA:7,0,2,0
BRDA:7,0,3,1
BRDA:7,0,4,-
BRDA:7,0,5,-
BRDA:7,1,0,- (the <block number> changed from 0 to 1 after merging two info files if linux info file is after windows info file)
BRDA:7,1,1,-
BRDA:7,1,2,0
BRDA:7,1,3,1

The example code is very simple, but suppose we have complex conditions in real project which the branch in line:7 are multiple conditions and some of them are not covered (neither some platform-specific template instantiations nor code logic depends on different platforms). Then, we plan to write more test cases to improve coverage. But when we run test cases to generate new coverage data, how can we know which branch coverage got improvement? BRDA:7,1,2,0 or BRDA:7,0,2,0.

[henry2cox]

The first integer after "BRDA:" is the source line number.
All your numbers are "7" 0 for lines 16, 25, and so forth.

Your compiler/toolchain is broken and is generating bogus branch information.

You should try to upgrade to a newer version - and see if that addresses the issue.
Or move to LLVM or GCC. Both work better.

[myler]

All your numbers are "7" 0 for lines 16, 25, and so forth.

Sorry, this makes confusion, I just removed unrelated output, only kept line: 7 here.
As you can see, the second field <block number> changed from 0 to 1 for appended .info file.

[henry2cox]

Your first block (numbrerd 0) has 6 branch tokens.
Your second has 4.
They are not the same - so they don't get merged.

The block ID is not meaningful - except block IDs on a particular line are unique.
When merging: lcov will look at a block containing 4 branch tokens and attempt to merge it with a block from the other DB which also has 4 branch tokens (if the branch expressions are identical. For gcov generated branch data: there is no branch expression - just an integer visit order index. Those are going to be identical for any block containing 4 tokens.)

Rather than hacking up data manually and trying to illustrate some point, it is far better - and far easier for both of us - to instead show a complete example (with source code and makefiles). Then we know that we are both looking at valid, tool-generated data, and that we are both seeing the same issues.

[myler]

Let me show complete example:

Environment:

Linux OS: Red Hat Enterprise Linux 9.0 (Plow)
Window OS: Windows Server 2025 Standard 24H2

Compiler version:

Linux

> clang++ --version
clang version 21.0.0git (3215fd70294369b73b0e944238097a28550e7435)

Windows

> clang-cl --version
clang version 21.0.0git (3215fd70294369b73b0e944238097a28550e7435)

llvm-cov:

> llvm-cov --version
LLVM (http://llvm.org/):
  LLVM version 21.0.0git

> cat llvm-gcov.sh
llvm-cov gcov "$@"

lcov version:

> lcov --version
lcov: LCOV version 2.2-beta

Attached gcov_example.tar.gz
Please unpack it by tar -xzf gcov_example.tar.gz and see .gcno, .gcda, test.cpp, llvm-gcov.sh, and .info files.

Build commands

Linux:

> clang++ -coverage test.cpp
> ./a.out
> ls
a.out a-test.gcda  a-test.gcno  test.cpp

Windows:

> clang-cl --version
clang version 21.0.0git (3215fd70294369b73b0e944238097a28550e7435)

> clang-cl --coverage test.cpp
> test.exe
> dir /b
a-test.gcda
a-test.gcno
test.cpp
test.exe

# scp a-test.gcda a-test.gcno test.exe to Linux `shared` directory and run lcov to generate `windows.info`

Generate tracefile

In order to keep SF: as same to avoid running on different path of different machines, we copy gcov data in share directory before running lcov.
Linux

> cp lin/a-test.gcda  lin/a-test.gcno share/
> cd share
> lcov --capture --directory . --branch-coverage --ignore-errors source,inconsistent --no-external --gcov-tool ./llvm-gcov.sh -o ../linux.info

Windows

> rm share/a-test.gcda share/a-test.gcno
> cp win/a-test.gcda  win/a-test.gcno share/
> cd share
> lcov --capture --directory . --branch-coverage --ignore-errors source,inconsistent --no-external --gcov-tool ./llvm-gcov.sh -o ../windows.info

Merge tracefiles

> lcov --branch-coverage  --gcov-tool ./llvm-gcov.sh -a ./linux.info -a ./windows.info -o merged_lin_win.info
> lcov --branch-coverage  --gcov-tool ./llvm-gcov.sh -a ./windows.info -a ./linux.info -o merged_win_lin.info

[myler]

Hello @henry2cox,

Recently, I got more clues for this issue. Please consider following use case:

A very simple header file branch.hpp

#include <iostream>

template <typename T>
void example(T value) {
    if (value > 0) {
        std::cout << "Value: " << value << " is greater than 0." << std::endl;
    }
}

Two cpp files include this header file: branch_1.cpp and branch_2.cpp

#include "branch.hpp"

void branch_1() {
    example<int>(1);
    example<unsigned int>(1);
    example<long>(1);
}

#include "branch.hpp"

void branch_2() {
    example<bool>(true);
    example<char>('a');
}

When we use lcov to merge the info file of branch_1.cpp and branch_2.cpp, it is order sensitive.

lcov --branch-coverage -a branch_1.info -a branch_2.info -o merge_1_2.info
lcov --branch-coverage -a branch_2.info -a branch_1.info -o merge_2_1.info

I also see gcov has a option --long-file-names :

--long-file-names
Create long file names for included source files. For example, if
the header file x.h contains code, and was included in the file
a.c, then running gcov on the file a.c will produce an output file
called a.c##x.h.gcov instead of x.h.gcov. This can be useful if
x.h is included in multiple source files and you want to see the
individual contributions. If you use the -p option, both the
including and included file names will be complete path names.

Is it possible to provide an option that can sort the block merge order by alphabetical order of this information?

[myler]

Thank you so much for this feature, I downloaded the latest version and ran the lcov command with --sort-input successfully.

When I use gcov tool with -l or --long-file-names, I can generate two gcov files for single header file because it is included in two source file, let's reuse above example:
Perform llvm-cov gcov -b -l branch_1.cpp, branch_1.cpp##branch.hpp.gcov can be generated.
Perform llvm-cov gcov -b -l branch_2.cpp, branch_2.cpp##branch.hpp.gcov can be generated.
(If I perform gcov wthout -l, the gcov output file for this header would be branch.hpp.gcov and it always be overwitted by the last command)

So if I use lcov -c -d . --gcov-tool <my full path of gcov tool> -o branch.info --branch-coverage --sort-input. There is a single SF:<my full path of branch.hpp> in the info file.
Do you know how to let lcov or the detail command geninfo to generate separated SF: for header file like gcov tool?

SF:<my full path>/branch_1.cpp##branch.hpp
...
SF:<my full path>/branch_2.cpp##branch.hpp
...

[henry2cox]

I don't think I understand your use model/I am not quite sure what your end goal is - but I fear that that is not how lcov works.
The intent is to aggregate the coverage data across tests (and potentially across aliases created by the build environment) and through time.

It is possible to collect per-testcase coverage data - and you could possibly use that feature to collect per-compilation-unit data (at the cost of serial execution). This doesn't come for free, of course. Per-testcase data requires the DB to hold nCoverPoints * nTests elements - which will become large, rather quickly.

If you really want per-unit coverage data, then perhaps lcov is not the tool for you.

[myler]

My goal is not to collect per-testcase (I agree with you that this will generate huge data and requires the DB)
Let me show the real case (not example) in my production environment (I verified the commands on RHEL9.0 with system gcc):

# clone latest opensource llvm
git clone https://github.com/llvm/llvm-project.git

# build and install llvm with default system compiler (gcc/g++)
cd llvm-project
cmake -S llvm -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=`pwd`/install -DLLVM_ENABLE_PROJECTS=all | tee cmake.log
cmake --build build -j32 | tee build.log
ninja -C build install | tee install.log

# build instrumented llvm by prebuilt clang/clang++ (self-build)
export LOCAL_CLANG_ROOT=`pwd`/install
export PATH=$LOCAL_CLANG_ROOT/bin:$PATH
export CPATH=$LOCAL_CLANG_ROOT/include:$CPATH
export LIBRARY_PATH=$LOCAL_CLANG_ROOT/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=$LOCAL_CLANG_ROOT/lib:$LD_LIBRARY_PATH
cmake -S llvm -B self_build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=`pwd`/self_install -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;lld' -DCMAKE_C_COMPILER=`pwd`/install/bin/clang -DCMAKE_CXX_COMPILER=`pwd`/install/bin/clang++ -DCMAKE_C_FLAGS="--coverage" -DCMAKE_CXX_FLAGS="--coverage" | tee self_cmake.log
cmake --build self_build -j32 | tee self_build.log
ninja -C self_build install | tee self_install.log

# run some tests to generate .gcda data
export LOCAL_CLANG_ROOT=`pwd`/self_install
export PATH=$LOCAL_CLANG_ROOT/bin:$PATH
export CPATH=$LOCAL_CLANG_ROOT/include:$CPATH
export LIBRARY_PATH=$LOCAL_CLANG_ROOT/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=$LOCAL_CLANG_ROOT/lib:$LD_LIBRARY_PATH
ninja -C self_build check-all | tee check-all.log

# lcov
lcov --branch-coverage --capture --all -f -d . --gcov-tool ../llvm-gcov.sh --ignore-errors source,gcov,graph,inconsistent,unused,path,range -o check-all.info
genhtml --branch-coverage --no-function-coverage --ignore-errors source,gcov,graph,inconsistent,unused,path,range --filter missing check-all.info -o check-all.html

# review code coverage report
cd check-all.html
python3 -m http.server
# open browser and direct the link http://0.0.0.0:8000/clang/include/clang/AST/RecursiveASTVisitor.h.gcov.html

Let me use line: 1113 as example:

File clang/include/clang/AST/RecursiveASTVisitor.h, in the template class RecursiveASTVisitor, there is a macro DEF_TRAVERSE_TYPE(TYPE, CODE):

And branches are introduced by the if statements in this macro, so we got branch coverage result like this:

With the --sort-input, I think the order of block can be fixed, but I still have no idea about how to identify what type of branches taken or not taken in template class of common header files.
As you may know, llvm is a well-known project, and many products are derived from it. So, I think this is a very common problem.

[henry2cox]

Yeah...one of the fun things about macros is that they are all on one line.
One rather hacky approach (which is also not easy) is to explicitly generate the -E preprocessor output, then munge the code to break up the multi-line macros - and then generate coverage on the hacked file.
I think that will be more difficult than it appears on the surface...and I suspect that yet more issues that we haven't foreseen will raise their ugly little heads.

On the other hand: if you have the GCDA data for a particular compilation unit, then you could extract coverage for that unit by itself and then munge the LCOV-generated coverage data file to encode information into the 'branch expression' part of the data - say, if the data looked like:

SF: my/shared/header/file
VER:whatever
...
BRDA:1114,0,1,10  <- line 1114, block 0, branchID 1, taken 10 times
...

to something like:

SF: my/shared/header/file  <- same
VER:whatever  <- same
...
BRDA:1114,0,compile_unit_identifier:1,10  <- line 1114, block 0, branch expression "compile_unit_identifier:1", taken 10 times

(and the same for MC/DC entries).

Post munging, when lcov goes to aggregate the coverage data, the branch expressions won't match for different instantiations of your header file and so won't be merged - and the ID information will appear in the corresponding HTML popup.
The same compilation unit hack will work exactly the same way for languages that do annotate actionable branch expression strings.

One significant down side is that there are a lot of files if you instrument all of LLVM - and running that will take a LONG time, if you have to process serially. (Takes about 20+ minutes on a substantial server, for us -when we exploit as much parallelism as we can.)
OTOH: if this would work and and turns out to be useful, it could be built into the tool (as yet another callback).
Another issue is that you want this sort of speciation in only certain locations. Generally, you do want to merge coverage data across compilation units.

[myler]

One rather hacky approach (which is also not easy) is to explicitly generate the -E preprocessor output, then munge the code to break up the multi-line macros - and then generate coverage on the hacked file.

I did some experiments on example code with macro.

Even if I use clang -E to generate .ii file, the multi-line macro will be expanded to single-line statement because these \ at the end of each line inside the macro are treated as one line. And when I perform lcov --capture ..., the SF: still point to header file not .ii file.

encode information into the 'branch expression' part of the data

This works for me with manual inserting some type information, then I need to figure out how to get compile_unit_identifier or also demangled function name for the branch?

there are a lot of files if you instrument all of LLVM - and running that will take a LONG time

Actually, compared to the time it takes to compile a large number of test cases using this instrumentation compiler, the time spent using the lcov is not too long for me.
It is not easy to instrument part of LLVM, because the clang compiler driver invokes many toolsets in the backend, e.g., cc1, opt, llvm-*, etc, and such common header file RecursiveASTVisitor.h may be included into various object files that are linked to these tools.

[henry2cox]

Yep. For macros, you need to break up the lines you care about, so all the statements don't have the same address.
And you have to munge the #line directives (not all of them...just the ones you care about).

If the issue is common headers and you want data from only one (or some) specific tools, then one approach is to configure two build areas - one instrumented for coverage, one not.
Compile both of them - then swap the instrumented executables into the opt build. Shared libraries are going to be a bigger issue.

With respect to munging lcov data: the GCDA file corresponds to the compilation unit.
If you are thinking about transitively included headers (not directly included into the cpp file) - then yeah...harder. You are going to need a tool to reconstruct the physical layout of the code (not easy - but you can see it in the -E output).
If you have the compilation unit data and you know which lines (which macros) you care about: then those are the only BRDA records whose branchExpr you need to munge. From your problem statement: I think you want to know which branch records on this line correspond to file A, file B, etc - and both keep the data for each file separate, but also to aggregate coverage data across testcases.

[myler]

You are going to need a tool to reconstruct the physical layout of the code

This attempt may be less effort for me, after all, code coverage collection during compilation and testing take more time, especially the collection data per compilation unit.
Maybe, to duplicate header files included by each cpp file by renaming as <source file name>##<header file name> before build.
And munge SF: and branchExpr in the info file, e.g.:

SF:my/shared/header/cpp1##file
...
BRDA:1114,0,1,10  <- line 1114, block 0, branchID 1, taken 10 times
...
SF:my/shared/header/cpp2##file
...
BRDA:1114,0,1,20  <- line 1114, block 0, branchID 1, taken 20 times
...

to something like:

SF:my/shared/header/file
...
BRDA:1114,0,cpp1:1,10  <- line 1114, block 0, branchID 1, taken 10 times
BRDA:1114,0,cpp2:1,20  <- line 1114, block 0, branchID 1, taken 20 times
...