clang source-based code coverage *sometimes* incorrectly includes comments into coverage reports

[krinkinmu]

Hi There,

I believe I found a bug in the clang source based coverage. It manifests for me on the following clang version:

clang++ --version
Ubuntu clang version 18.1.8 (++20240731024944+3b5b5c1ec4a3-1~exp1~20240731145000.144)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

I posted the problem on LLVM discussions (see https://discourse.llvm.org/t/llvm-source-based-coverage-sometimes-generates-counter-for-comments/85730), however since then I found a sensibly small reproducer for the issue, so I'm reporting it as a bug now.

Here is C++ snippet I use to reproduce the issue:

#include <iostream>
#include <memory>

template <typename RequestType, typename ResponseType> class Stream {
public:
  virtual ~Stream() = default;
  virtual void onEvent() = 0;
};

template <typename RequestType, typename ResponseType>
class StreamImpl : public Stream<RequestType, ResponseType> {
public:
  void onEvent() override {
    // A comment that should not be counted by coverage.
    std::cout << "onEvent\n";
  }
};

using Request = int;
using Response = int;

using Processor = Stream<Request, Response>;
using ProcessorPtr = std::unique_ptr<Processor>;

inline ProcessorPtr createSomething() {
  return std::make_unique<StreamImpl<Request, Response>>();
}

class TestLibrary {
public:
  TestLibrary();
  ~TestLibrary();
  void method();
};

TestLibrary::TestLibrary() = default;
TestLibrary::~TestLibrary() = default;

void TestLibrary::method() {
  std::cout << "TestLibrary::method()\n";
}

int main() {
  TestLibrary library;
  library.method();
  return 0;
}

For completeness here are the commands used to generate coverage:

clang++ -fprofile-instr-generate -fcoverage-mapping test.cc -o test
LLVM_PROFILE_FILE="prof.raw" ./test
llvm-profdata merge -sparse prof.raw -o prof.data
llvm-cov export --format=lcov ./test -instr-profile=prof.data

The re-producer is somewhat convoluted, but what I want to concentrate on here is coverage of StreamImpl::onEvent method and specifically the line with comment in that method (line 14).

There is a range of results that, I think, are sensible here:

1. StreamImpl::onEvent is not included in the coverage at all - it's not used, it's a method of a template that does not need to be instantiated in principle (createSomething function isn't used and there is no other places where this template is used).
2. StreamImpl::onEvent could be included in the coverage report with 0 coverage, but the comment on line 14 should not be included in the coverage report (e.g., we should not report 0 coverage for comments, instead we should not report coverage for comments at all).

However, what I actually get is the following:

FN:36,_ZN11TestLibraryC2Ev
FN:37,_ZN11TestLibraryD2Ev
FN:39,_ZN11TestLibrary6methodEv
FN:43,main
FN:25,_Z15createSomethingv
FN:13,_ZN10StreamImplIiiE7onEventEv
FNDA:1,_ZN11TestLibraryC2Ev
FNDA:1,_ZN11TestLibraryD2Ev
FNDA:1,_ZN11TestLibrary6methodEv
FNDA:1,main
FNDA:0,_Z15createSomethingv
FNDA:0,_ZN10StreamImplIiiE7onEventEv
FNF:6
FNH:4
DA:13,0
DA:14,0
DA:15,0
DA:16,0
DA:25,0
DA:26,0
DA:27,0
DA:36,1
DA:37,1
DA:39,1
DA:40,1
DA:41,1
DA:43,1
DA:44,1
DA:45,1
DA:46,1
DA:47,1
BRF:0
BRH:0
LF:17
LH:10

As you can see there is a 0 reported for line 14 which is not what I expect to see.

If I change the code to remove the inline modifier from createSomething function, I get a reasonable lcov:

FN:25,_Z15createSomethingv
FN:36,_ZN11TestLibraryC2Ev
FN:37,_ZN11TestLibraryD2Ev
FN:39,_ZN11TestLibrary6methodEv
FN:43,main
FN:6,_ZN6StreamIiiED2Ev
FN:13,_ZN10StreamImplIiiE7onEventEv
FNDA:0,_Z15createSomethingv
FNDA:1,_ZN11TestLibraryC2Ev
FNDA:1,_ZN11TestLibraryD2Ev
FNDA:1,_ZN11TestLibrary6methodEv
FNDA:1,main
FNDA:0,_ZN6StreamIiiED2Ev
FNDA:0,_ZN10StreamImplIiiE7onEventEv
FNF:7
FNH:4
DA:6,0
DA:13,0
DA:15,0
DA:16,0
DA:25,0
DA:26,0
DA:27,0
DA:36,1
DA:37,1
DA:39,1
DA:40,1
DA:41,1
DA:43,1
DA:44,1
DA:45,1
DA:46,1
DA:47,1
BRF:0
BRH:0
LF:17
LH:10

NOTE: The reason why inline is there is that originally that function was defined in a header file.

[EugeneZelenko]

Could you please try 20 or main branch?

[krinkinmu]

@EugeneZelenko got the same result on the following version:

clang++-20 --version
Ubuntu clang version 20.1.2 (++20250401083310+e7406753caf3-1~exp1~20250401203329.93)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin

[krinkinmu]

I have a question to clarify the situation and context around source-based coverage in Clang.

For the background, there is also another issue with coverage that bugs us somewhat (#32849) which results in coverage reports randomly producing wrong results, which undermins a bit the use of coverage reporting in our CI.

I've asked if there is anything I could help with on the original issue, but didn't get any response and now we hit this issue, which seems somewhat related. Given that I couldn't help but start wondering if source-based coverage in Clang has any active maintainers working on it?

I'm not asking to put blame on anybody, just want to get an idea of the situation to see if:

1. there is something specific I can do to help (granted I don't really have any experience with Clang or LLVM internals or related tools, like cov/lcov)
2. we can rely on the source-based coverage in Clang going forward or should start looking for alternatives.

[hvdijk]

Exporting without --format=lcov explains what's going on: it includes "regions":[[13,27,16,4,0,0,0,0]]. The first four numbers here are starting line, column, ending line, column. What this is indicating is that the complete function body, from opening { to closing }, is treated as a single region to be counted. The results you get are reliable, it's not actually recording that the comment is executed, but the results are very confusing when converted to lcov, it sure makes it look like the comment is executed.

Clang should not be including those braces in the region, IMO, but hopefully knowing that it does this, you can make better sense of the output you get.

[krinkinmu]

@hvdijk I get the explanation, but I think you're overlooking practical consequences of the current behaviour here.

A very small change in the code results in a completely different coverage report even though it should change nothing. So if we take the coverage reports as they are now and put them into an automatic CI that provides a simple test coverage check, we end up with a very fragile test that could be broken by a completely benign changes to the code. That's very bad for CI and it undermines the trust into tests the CI does. That basically teaches people to ignore CI checks because they now that they can be broken,

That's exactly what I mean when I say that the results are not reliable - we can't build a stable and predicatble automatic check based on those results.