Skip to content

ci: scope the C++ coverage report and add per-directory rows - #29

Merged
Gajesh2007 merged 5 commits into
mainfrom
ci/coverage-scope
Oct 2, 2026
Merged

Gajesh2007 merged 5 commits into
mainfrom
ci/coverage-scope

Conversation

@davidtai

@davidtai davidtai commented Sep 29, 2026 •

Copy link
Copy Markdown

Proposed changes

The C++ coverage step of Fork Tests now says what its number measures. The ruling (David Tai, 2026-09-27 and 2026-09-29): the target is 80% line coverage, reported and never enforced, measured on host code only.

  • The job summary shows one table: Component, Lines, Branches, Functions, Target. One row covers all host C++ under mlx/. One row each covers mlx/backend/metal, mlx/backend/cpu, mlx/backend/common, mlx/distributed and mlx/io. The step runs llvm-cov report once for each row, with the directory as the source path.
  • The ignore regex keeps every pattern from ci: report C++ line coverage from the Metal test run (re-land of #25) #28 (tests, Python bindings under python/, build folders, virtual environment and Python headers). It adds mlx/backend/cuda/ (the CUDA backend, which macOS builds only as a stub) and mlx/3rdparty/ (the vendored pocketfft header). The Metal shader source needs no pattern: the host compiler never builds it, so it has no coverage records. The few headers in mlx/backend/metal/kernels/ that host C++ includes, such as steel/conv/params.h, stay counted.
  • Text under the table says what is in the denominator, what is left out and why, that the Branches column is real C++ branch coverage, and that the target is reported only.
  • The step reads each value from the TOTAL row by its column name in the header row, not by field position. The step fails when a value is missing, or when a directory does not exist (llvm-cov otherwise ignores the path and reports every file). The Python table and the llvm-cov export call that fed it are gone.

Nothing else in the job changes. fork.yaml lists .github/workflows/fork-tests.yml in the "Fork tooling" section, so no fork.yaml change is needed.

Test plan

Job summary of that run, as the step wrote it (values checked against the TOTAL rows in the step log):

C++ line coverage

Test step: success

Component Lines Branches Functions Target
All of mlx/ 70.08% 62.69% 72.64% 80%
mlx/backend/metal 81.34% 66.62% 82.24% 80%
mlx/backend/cpu 45.89% 39.21% 56.80% 80%
mlx/backend/common 89.37% 84.03% 93.04% 80%
mlx/distributed 23.70% 15.23% 28.94% 80%
mlx/io 83.60% 80.31% 95.24% 80%

The denominator is the host C++ code under mlx/ that the C++ tests and the Python wheel link: 50737 lines. It leaves out the tests, the Python bindings in python/src, the build folders with their dependencies and generated files, the vendored mlx/3rdparty/pocketfft.h, and the CUDA backend in mlx/backend/cuda/, which macOS builds only as a stub.
The Metal shader source in mlx/backend/metal/kernels/ is not in the denominator: the kernels run on the GPU and are outside every coverage tool. The host code that encodes and dispatches them is counted, and so are the few headers in that folder that host C++ includes.
The Branches column is real C++ branch coverage. The 80% target is reported only. The job does not fail below it.

Before this PR (#28 head, run 36505380323) the same job counted 53027 lines at 68.26%. The difference is the CUDA stub (9 lines) and the pocketfft header (2281 lines). steel/conv/params.h (17 lines) stays counted.

CONTRIBUTING checklist

  • Formatting: not applicable, only a workflow file changes; actionlint passes on it.
  • Tests: not applicable, no library code; the Fork Tests run of this PR is the test.
  • Benchmarks: not applicable, no library code changes.

Dependencies

#28 is not merged yet. This branch is based on ci/coverage-on-main (#28's head, ec82b43), so the diff against main shows #28's two commits too. Merge #28 first; this PR then adds one commit. If #28 is squash-merged, rebase this branch with git rebase --onto main ec82b436.

Authorship

Written by Claude (Opus 5.5) under David Tai's direction.

  • ☐ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: this change and this description were written by Claude (Opus 5.5) under David Tai's direction.

🤖 Generated with Claude Code

davidtai and others added 4 commits September 28, 2026 19:53
Build with clang coverage instrumentation through CFLAGS and CXXFLAGS.
Both builds in build-macos run cmake from scratch and read them. The
test step writes one profile for each process and instrumented binary.

After the tests, merge the profiles and report line coverage for the C++
test binary and the installed Python extension and libraries. Write the
totals to the job summary and upload the full report as an artifact.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first report counted the generated sources in the two build
folders and the headers of the uv Python. Ignore every path under a
build folder and the uv Python folder, so that the totals count only
the mlx sources.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The job summary now shows one table with Lines, Branches and Functions for
all host C++ under mlx/ and for backend/metal, backend/cpu, backend/common,
distributed and io, each against the 80% target. The target is reported
only.

The report now also leaves out the Metal shader source in
mlx/backend/metal/kernels/, the CUDA backend and the vendored pocketfft
header. The step reads each value from the llvm-cov TOTAL row by its
column name and fails when a value is missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernels exclude dropped steel/conv/params.h, a header that host C++
includes and that has coverage records. The shader source has no records,
so the exclude is removed. The text says "link" instead of "run", and a
comment notes that value_of rejects the "-" of a column without counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	.github/workflows/fork-tests.yml
@Gajesh2007
Gajesh2007 merged commit bfa373f into main Oct 2, 2026
6 checks passed
@davidtai
davidtai deleted the ci/coverage-scope branch October 4, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants