Skip to content

feat: add threshold flags for CI regression gating - #6

Open
mvanhorn wants to merge 1 commit into
entireio:mainfrom
mvanhorn:fix/forgemark-threshold-gate
Open

feat: add threshold flags for CI regression gating#6
mvanhorn wants to merge 1 commit into
entireio:mainfrom
mvanhorn:fix/forgemark-threshold-gate

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

forgemark can now gate CI runs: set -min-push-rate, -max-p95, or -max-errors and a sweep that misses any of them prints per-level breach lines to stderr and exits 2. All three flags default off; unset flags change nothing.

Why this matters

The results JSON is written "for charting" (README), but the process itself always exits 0 - run() returns writeResults(...) without inspecting the measured levels, so a deploy that halves push throughput still reads as a passing benchmark. Both reference tools in this space gate on results: GitLab Performance Tool ships built-in success thresholds on throughput and latency, and k6 fails the process on a threshold breach. This adds the equivalent with three stdlib flags and a distinct exit code (0 ok, 1 operational error, 2 threshold breach), so a forge team can pin a floor on push/s and a ceiling on p95 in CI.

Demo

Simulated demo:
demo

Verified live against github.com: a c=1 run sustaining 1.0 push/s with -min-push-rate 5 -max-p95 500ms printed both breach lines and exited 2; the same run with -min-push-rate 0.2 -max-p95 5s exited 0 with "thresholds_ok": true in the JSON.

Changes

  • Threshold evaluation is a pure function over the existing levelResult fields (checkThresholds in stats.go), run after writeResults so the JSON is always written even when the run fails the gate.
  • Exit 2 is reserved for breaches via a sentinel error in main(), keeping it distinguishable from the hard-error exit 1 path.
  • The thresholds block (configured values plus breach list) only appears in the results JSON when at least one flag is set; unset runs produce byte-identical output.

Testing

Table-driven TestCheckThresholds (breach / no-breach / unset), TestParseFlagsThresholds, and two writeResults JSON tests (thresholds present when configured, absent when unset). go test ./..., go vet, gofmt, and golangci-lint all pass locally.

Add -min-push-rate, -max-p95 and -max-errors. When any is set and a
measured level misses it, forgemark prints per-level breaches to stderr,
records them in the results JSON, and exits 2 (distinct from 1 for
operational errors). All three default off.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant