feat(l4): add l4.measurement-backed — the autonomy-without-guardrails check

[sroberts]

The second of the two anti-patterns CLAUDE.md says the assessor should flag. The L3 one — the dashboard graveyard — got l3.metrics-acted-on in #44. This is its counterpart, and it completes the pair.

Design: same inversion as the L3 one

Status: Found means "present" everywhere in the catalog, so a detector that fires on something bad would raise the level score by finding it. This asks the positive question — is the autonomy in this repo backed by a gate that can stop it? — and returns Missing for the anti-pattern. NA when there's no repo-modifying automation, which excludes it from the level average per §7.

Detection

The paper's claim is that levels are sequential: L4 without L3 isn't "most of the way to L4", it's a machine committing to a codebase nothing is checking. Asked in the form a static scan can answer:

• Autonomy — a step using create-pull-request, an automerge action, github-script, or running git push / gh pr merge / gh issue create.
• Guardrail — a push/pull_request-triggered workflow containing something that can fail: exit 1, ::error::, a test runner, a linter action.

Autonomy with no guardrail → Missing. Both → Found. No autonomy → NA.

Scheduled workflows deliberately don't count as guardrails. A nightly suite reports; it doesn't block a merge. Counting one would pass exactly the repos this signal exists to catch — automation running against a codebase whose only checks happen after the fact. There's a test pinning that behaviour, because it's the kind of thing a later "improvement" would quietly relax.

What a pass does not mean

Both caveats ship in the result's notes, not just in SPEC.md, since the notes are what someone reads when they see a green result:

• It doesn't establish that the gate covers the code path the automation touches.
• It doesn't establish that branch protection requires the check to pass. Branch protection is repo configuration, not filesystem state, and plumbline doesn't read it.

Worth stating plainly given this repo's own history: main here has no branch protection at all, so every gate in ci.yml is advisory. This signal still returns Found for plumbline — correctly by its rule, and incompletely by any stricter reading. Confidence is capped at medium for exactly this reason.

The drift gate earned its keep

The skill-body test from #44 failed on the first full run of this branch, unprompted:

--- FAIL: TestSkillBodyListsEverySignal
    signals registered but absent from the skill body's stable-contracts list: [l4.measurement-backed]

Second thing it's caught, both times before review.

Effect on plumbline's own score

L4 goes 0.8 → 0.833; verdict stays level 5. docs/SIGNALS.md and .plumbline.toon regenerated (signals[23]). Full suite, go vet, gofmt clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XtNwLDzRYmPrXcYeQypTf9

[Copilot]

Pull request overview

Adds a new Level 4 (“Adaptive”) signal to Plumbline’s ACMM catalog to detect the autonomy-without-guardrails anti-pattern by checking whether repo-modifying automation is accompanied by a push/PR-triggered workflow that can fail (scheduled workflows explicitly don’t count as guardrails).

Changes:

• Introduces l4.measurement-backed detector (internal/signals/l4/measurement_backed.go) and registers it in the default signal set.
• Adds unit tests covering NA / Missing / Found behavior and the “scheduled suites don’t count” rule.
• Updates the signal catalog documentation and the embedded skill “stable contracts” list to include the new signal.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
SPEC.md	Documents the new L4 signal and its intended semantics/limits in the catalog spec.
internal/skill/skill.go	Adds measurement-backed to the stable list of L4 signal IDs.
internal/signals/l4/measurement_backed.go	Implements the new detector and registration.
internal/signals/l4/l4_test.go	Adds test coverage for the new detector, including the scheduled-guardrail exclusion.
docs/SIGNALS.md	Regenerates the signal catalog docs to include the new signal and updated count.
.plumbline.toon	Regenerates the repo’s snapshot output to include the new signal and updated L4 score.

Suppressed comments (1)

internal/signals/l4/measurement_backed.go:38

• gateUsesRE currently treats codecov and any sonarsource action as a blocking gate, but these are often metric publishers/scanners that don’t necessarily fail a PR/push. This can produce false Found results for “guardrails” when the workflow only uploads/reports. Consider restricting this to actions that are clearly used as failing lint/analysis gates (similar to l3.build-lint-gate’s action vocabulary).

	gateRunRE  = regexp.MustCompile(`(?m)(exit\s+1|::error::|go\s+test|pytest|npm\s+(run\s+)?test|cargo\s+test|--fail)`)
	gateUsesRE = regexp.MustCompile(`(?i)(golangci|staticcheck|codecov|sonarsource|reviewdog)`)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.