Creating this issue to capture the plan for coverage measurements. This will be a long journey, so we are taking a pragmatic approach by limiting the scope and intentionally including some temporary workarounds to unblock the functionality we need sooner. After this we can think about additional enhancements such as differential coverage for PRs and local changes, dashboard, and other quality of life enhancements.

• Coverage measurements
  • On-target profile data
    • Initial experiments, PoC, and foundation ([sw] Coverage support for OpenTitan #3080)
    • Enable coverage instrumentation using config settings and compiler flags as needed ([coverage] Refactoring for different measurement types #16947 , [coverage] Functional Test Coverage #16951)
      • This is a temporary measure to unblock progress until we can integrate our changes with bazel. Some resources: 1, 2, 3.
      • Coverage runtime ([coverage] Fixes and updates #16762)
      • Coverage instrumentation ([coverage, bazel] Refactoring for on-target coverage measurements #16944 , [coverage] Functional Test Coverage #16951)
        • Compiler flags for silicon creator targets ([coverage] Functional Test Coverage #16951)
        • Coverage measurements from functional tests ([coverage] Functional Test Coverage #16951)
          • Measured coverage of (daily) commits between 2022-12-20 - 2022-12-31.
        • Coverage measurements from ROM E2E tests
          • This requires some investigation since instrumented binary does not fit in 32 KiB when built with -O0. We must support coverage with optimization and be careful about what we are instrumenting.
          • Reduce coverage instrumentation size overhead.
          • Transfer profile buffer before ROM hands over execution (either to ROM_EXT or shutdown).
    • Coverage buffer transfer ([coverage] Fixes and updates #16762)
      • Mostly there, we should be more clear about endianness, increase buffer size, and break long lines to prevent buffer issues.
    • Update vendored llvm_clang_rt_profile to llvmorg-13.0.1 ([coverage] Fixes and updates #16762)
      • We collect raw profile data from our target and there are no cross-version compatibility guarantees.
    • Parse, index, and merge raw profile data in test logs ([coverage] Functional Test Coverage #16951)
    • Remove vendored repo and use bazel to pull llvm_clang_rt_profile ([bazel, coverage] Add repository rule for LLVM compiler-rt and remove the previously vendored copy from the tree #16973)
      • llvm-project releases a smaller archive that includes the part we need.
  • Off-target profile data (unit tests)
    • Bazel should support this out of the box. We can use this to unblock CI integration and work on it in parallel.
    • Depending on the profile data format that bazel uses we may need to convert to/from llvm profile data.
    • Bazel has only primitive support for clang's source-based code coverage. We need to merge the profile data, process object files and generate reports.
    • bazel config changes for coverage measurements using clang ([coverage] Add unit test coverage measurement support #16891)
    • build fixes for clang ([sw, bazel] Build and bazel target fixes for off-target (unit test) coverage #16890)
    • Add clang-{lowRISC RISC-V toolchain version} to opentitan container Dockerfile ([util/container] Add clang-13 to opentitan container #16888).
      • lowRISC RISC-V toolchain version is 13.0.1.
    • Determine the libraries that we are interested in (mostly silicon_creator modules, drivers, and rom code). ([coverage] Add unit test coverage measurement support #16891)
  • Reports
    • See here.
    • On-target
      • Functests (silicon creator modules and drivers)
      • ROM e2e tests
        • Also consider gdb-based e2e tests. Find out if it's possible to measure coverage in asm files.
      • Merged on-target coverage measurements
    • Off-target unit tests
    • Merged on-/off-target coverage measurements
    • Additional columns in the merged report with data from unit/func/e2e.
    • Additional reports such as mock and *_unittest.cc coverage.
    • Refactor current implementation for multiple coverage configs, e.g. queries in hjson logic in py.
      • I decided not to pursue this path in favor of better bazel integration because it turns into a "workflow orchestration" project which is essentially bazel.
    • Update landing page.
  • Report generation and publication
    • Public GCS bucket
      • sw-coverage under opentitan.org/ot-fpga-runner for now.
    • Bucket layout:
      • <head_timestamp>-<head_hash>/<coverage_type>/{merged.profiledata, merged.so, report.txt, html}
    • Index page
      • A minimalist page (gist) for now.
      • Should be updated as we update the layout of the GCS bucket to include additional report types.
      • Results table
        • Dynamically generated.
      • Trend chart
        • Added a line chart (chart.js). The index page looks pretty nice now.
    • Nightly runs
      • We can start working on this sooner using off-target coverage measurements.
      • Local workstation
        • Started running unit test coverage script and uploading results to GCS on 2022-12-21 (using gsutil for now).
      • CI
        • Build and Run
        • Authenticate to GCP with OIDC/WIF and publish
• Improve coverage and coverage infrastructure
  • Add dashboard pages to the repo
  • Unit tests for rom.c (See [test] Add tests verifying the ROM/ROM-ext don't touch peripherals they shouldn't touch #16022)
  • False positives possible?
    • What does llvm-cov do if we have the profile data but not the object file?
  • static inline change required by optimizations eliminates hash-mismatch errors. Check if still need to create the merged libraries.
  • Can we integrate with bazel and bazel coverage better?
    • @dmcardle's comment
    • @dmcardle's comment

cc @cfrantz @arunthomas @milesdai @drewmacrae @dmcardle