feat: add threshold flags for CI regression gating - #6
Open
mvanhorn wants to merge 1 commit into
Open
Conversation
Add -min-push-rate, -max-p95 and -max-errors. When any is set and a measured level misses it, forgemark prints per-level breaches to stderr, records them in the results JSON, and exits 2 (distinct from 1 for operational errors). All three default off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
forgemark can now gate CI runs: set
-min-push-rate,-max-p95, or-max-errorsand a sweep that misses any of them prints per-level breach lines to stderr and exits 2. All three flags default off; unset flags change nothing.Why this matters
The results JSON is written "for charting" (README), but the process itself always exits 0 -
run()returnswriteResults(...)without inspecting the measured levels, so a deploy that halves push throughput still reads as a passing benchmark. Both reference tools in this space gate on results: GitLab Performance Tool ships built-in success thresholds on throughput and latency, and k6 fails the process on a threshold breach. This adds the equivalent with three stdlib flags and a distinct exit code (0 ok, 1 operational error, 2 threshold breach), so a forge team can pin a floor on push/s and a ceiling on p95 in CI.Demo
Simulated demo:

Verified live against github.com: a c=1 run sustaining 1.0 push/s with
-min-push-rate 5 -max-p95 500msprinted both breach lines and exited 2; the same run with-min-push-rate 0.2 -max-p95 5sexited 0 with"thresholds_ok": truein the JSON.Changes
levelResultfields (checkThresholdsin stats.go), run afterwriteResultsso the JSON is always written even when the run fails the gate.main(), keeping it distinguishable from the hard-error exit 1 path.thresholdsblock (configured values plus breach list) only appears in the results JSON when at least one flag is set; unset runs produce byte-identical output.Testing
Table-driven
TestCheckThresholds(breach / no-breach / unset),TestParseFlagsThresholds, and twowriteResultsJSON tests (thresholds present when configured, absent when unset).go test ./...,go vet,gofmt, andgolangci-lintall pass locally.