Skip to content

Conversation

@rtfeldman
Copy link
Contributor

@rtfeldman rtfeldman commented Dec 31, 2025

Summary

  • Add parser coverage enforcement at 84% minimum threshold
  • Coverage runs in a separate workflow that doesn't block main CI
  • Coverage runs on macOS ARM where kcov works correctly

Technical Details

  • kcov uses LLDB on macOS (works with Zig binaries)
  • kcov's ptrace-based approach on Linux doesn't work with Zig binaries (reports 0%)
  • Coverage workflow runs in parallel with main CI
  • Coverage takes ~5 hours but doesn't block CI completion
  • Coverage reports are uploaded as artifacts

Test plan

  • zig build coverage passes locally with 84.69% coverage
  • Main CI passes without being blocked by coverage
  • Coverage workflow created and configured

🤖 Generated with Claude Code

@diffray-bot
Copy link

Changes Summary

This PR adds code coverage enforcement to the parser build system with an 84% minimum threshold, prevents coverage regression, and adds snapshot tests for operators (pizza |>, caret ^) and malformed number/string error cases.

Type: mixed

Components Affected: Build system (build.zig), Parser source documentation (NodeStore.zig, tokenize.zig), Test suite (snapshot tests)

Files Changed
File Summary Change Impact
/tmp/workspace/build.zig Add MIN_COVERAGE_PERCENT constant (84%), modify parseCoverageJson() to return f64, add coverage validation logic with meaningful error messages, and configure kcov to exclude HTML.zig and debug statements. ✏️ 🔴
/tmp/workspace/src/parse/NodeStore.zig Add documentation comment explaining that the debug function is excluded from coverage measurement. ✏️ 🟢
/tmp/workspace/src/parse/tokenize.zig Add two documentation comments explaining debug output blocks are excluded from coverage. ✏️ 🟢
...orkspace/test/snapshots/parse_pizza_operator.md Add snapshot test for pizza operator ( >) parsing and precedence handling.
...orkspace/test/snapshots/parse_caret_operator.md Add snapshot test for caret operator (^) parsing and right-associativity. 🟢
...test/snapshots/parse_malformed_binary_number.md Add snapshot test for error handling of malformed binary numbers (0b without digits). 🟢
...ce/test/snapshots/parse_malformed_hex_number.md Add snapshot test for error handling of malformed hex numbers (0x without digits). 🟢
.../test/snapshots/parse_malformed_octal_number.md Add snapshot test for error handling of malformed octal numbers (0o without digits). 🟢
...rkspace/test/snapshots/parse_unclosed_string.md Add snapshot test for error handling of unclosed string literals. 🟢
Architecture Impact
  • New Patterns: Coverage-driven development enforcement via build system, Snapshot testing for parser behavior validation
  • Coupling: Coverage measurement is now tightly coupled to the build process - any parser changes must maintain or improve test coverage.
  • Breaking Changes: Build will fail if parser code coverage drops below 84% - developers cannot merge coverage-reducing changes

Risk Areas: Coverage threshold is hardcoded - increasing it requires code changes, Excludes debug statements which may hide untested error paths in production debugging code, HTML.zig exclusion assumes it's truly a visualization-only utility with no critical logic

Suggestions
  • Consider making MIN_COVERAGE_PERCENT configurable via build parameter for flexibility
  • Document the rationale for excluding HTML.zig and std.debug.* from coverage
  • Verify that snapshot tests are comprehensive enough to cover the major operators and error cases

Full review in progress... | Powered by diffray

rtfeldman and others added 3 commits December 31, 2025 23:58
kcov's ptrace-based approach doesn't work reliably with Zig-generated
binaries on Linux (reports 0 lines). On macOS, kcov uses LLDB which works.

Developers should run `zig build coverage` locally before merging.
The enforcement at 84% minimum threshold still works locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
kcov cannot instrument musl binaries (it uses ptrace/DWARF which
expects native glibc). The default target on Linux is musl for
static linking, so coverage tests were building musl binaries
that kcov couldn't instrument (reported 0% coverage).

This fix explicitly uses a native glibc target for the coverage
test binaries, while leaving the main roc binary as musl for
static linking.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rtfeldman
Copy link
Contributor Author

Superseded by #8878 which includes additional fixes for coverage enforcement

@rtfeldman rtfeldman closed this Jan 1, 2026
rtfeldman and others added 2 commits January 1, 2026 10:39
The previous fix used an empty target query which may still
default to musl in some contexts. This explicitly specifies
.abi = .gnu to ensure glibc is used for coverage binaries.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rtfeldman rtfeldman reopened this Jan 1, 2026
rtfeldman and others added 8 commits January 1, 2026 11:45
Use ./src/parse prefix like TigerBeetle uses --include-path=./src.
Remove extra exclude flags that may be causing issues.
Temporarily lower threshold to 50% to verify kcov works.
Instead of using setExecCmd which wraps the test, build the test
binary first, then run kcov as a separate system command.

This is how TigerBeetle runs kcov:
  kcov --include-path=./src output_dir binary_path
When running kcov on a single binary (not merging), it creates a
subdirectory named after the binary: parse_unit_coverage/coverage.json
Also set threshold to 0 so we can see the output without failing.
kcov has compatibility issues with Zig binaries on Linux, but works
reliably on macOS. Run coverage on the macos-15 (ARM64) runner only.

- Remove Linux glibc workaround (no longer needed)
- Restore 84% coverage threshold
- Only enable coverage step for macOS in build.zig
The nix flake only includes kcov on Linux, so install it via brew
before running coverage on macOS ARM.
Coverage runs on macOS ARM where kcov works correctly (via LLDB).
kcov doesn't work on Linux with Zig binaries (ptrace issue).

Coverage workflow:
- Runs in parallel with main CI
- Does not block CI completion (takes 5+ hours)
- Uploads coverage report as artifact
- Has 6-hour timeout

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rtfeldman rtfeldman marked this pull request as ready for review January 2, 2026 09:10
@rtfeldman
Copy link
Contributor Author

Closing in favor of #8864

@rtfeldman rtfeldman closed this Jan 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants