Nothing checks the story-coverage figures in CONTRIBUTING.md, which is how they drifted

[vladimirrott]

Corrected 2026-09-03 at 60af0ad. This issue was filed on 2026-08-17. PR #242 landed the next day and fixed two of the three claims below. One is still wrong, and the guard the issue asks for still does not exist. The original text is kept underneath so the history reads straight.

CONTRIBUTING.md publishes story-coverage figures that nothing derives from the story files.

What is still wrong at 60af0ad

CONTRIBUTING.md:50 says:

The suite is 133 stories: 54 atomic + 79 Ubuntu. Every Debian-only action now has one.

CONTRIBUTING.md:47, four lines above it in the same table, says the opposite:

Remaining: story coverage for the cross-family actions, and five Debian-only ones that still have none: the four fail2ban actions and GrubSetKargs.

Line 47 is the true one. Recounted at 60af0ad:

$ for a in Fail2banStatus Fail2banBanIp Fail2banUnbanIp ConfigureFail2banJail GrubSetKargs; do
    echo "$a: $(grep -rl "\"$a\"" tests/e2e/stories/ | wc -l)"
  done
Fail2banStatus: 0
Fail2banBanIp: 0
Fail2banUnbanIp: 0
ConfigureFail2banJail: 0
GrubSetKargs: 0

The story totals and the uncovered counts are correct today. Both recounted at 60af0ad:

$ ls tests/e2e/stories/story-*.sh | wc -l
133
$ # family from each story's own header line
ubuntu=79 atomic=54

and the derivation below now returns {'All': 59, 'Fedora': 10, 'Ubuntu': 5}, which is what CONTRIBUTING.md:50 prints.

So the numbers are right and nothing keeps them right. They were wrong for a day in August, somebody fixed them by hand, and the next story added or retired puts them back out.

The guard, which is the part worth doing

scripts/check_evidence_claims.py has no rule that derives an uncovered-action count. Its checks at 60af0ad:

$ grep -nE '^def check_' scripts/check_evidence_claims.py
124:def check_figure
149:def check_bare_test_totals
194:def check_story_claims
261:def check_bare_story_counts
306:def check_action_figures
382:def check_validated_tiers

check_bare_story_counts matches \b([0-9]{2,})[-\s]stor(?:y|ies)\b, so it catches "133 stories" and misses "59 are still untouched by any story" and "54 atomic". The guard is sound; the claim shape slips past it.

Scope

• Fix the contradiction: CONTRIBUTING.md:50 still claims every Debian-only action has a story.
• Add a check_evidence_claims.py rule that derives the uncovered counts and the family split from the tree, so neither can drift again. The derivation in the original text below is the whole implementation.

Tests first

Feed the checker a CONTRIBUTING.md fixture carrying a wrong count and assert it fails naming both figures, the published one and the derived one. A test that only exercises the agreeing case proves nothing, since that is the state the file is in today. Prove the rule bites by changing the published number and watching the checker go red, and again by adding a story file and watching it go red the other way.

Difficulty

medium. The documentation half is a two-line edit; the guard rule is the work.

---

Original text, filed 2026-08-17, before #242

CONTRIBUTING.md publishes two coverage figures and one prose claim about story coverage. All three are wrong.

The prose claim

Remaining: story coverage for the cross-family actions. The Debian-only ones are all covered now.

Five Ubuntu-family actions have no story: Fail2banStatus, Fail2banBanIp, Fail2banUnbanIp, ConfigureFail2banJail and GrubSetKargs. The fail2ban four are the subject of #219, which exists because of this gap.

The figures

of the action names available on both families, 57 are still untouched by any story, plus 8 Fedora-only ones.

Derived from the catalogue and the story files:

family	uncovered	CONTRIBUTING says
All (both families)	59	57
Fedora only	10	8
Ubuntu only	5	not mentioned
total	74	65

Derivation, from a clean checkout:

python3 - <<'PY'
import re, glob
rows = re.findall(r'^\| `([A-Za-z0-9_]+)` \|.*?\| (All|Ubuntu|Fedora) \|',
                  open('docs/action-reference.md').read(), re.M)
named = set()
for f in glob.glob('tests/e2e/stories/*.sh'):
    named |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
from collections import Counter
print(Counter(d for n, d in rows if n not in named))
PY

Two caveats a fixer should keep: this counts an action as covered if its name appears as a quoted literal anywhere in a story, which over-counts, because ConfigureWifi, GrubSetKargs and SetLocale appear only in "must not confuse with" comments. And covered-by-any-story is not the same as exercised-on-Ubuntu: 98 of the 164 Ubuntu-and-All actions are named by no ubuntu-family story.

Why the guard missed it

CONTRIBUTING.md is in CLAIM_FILES (scripts/check_evidence_claims.py), so this is not an unguarded file. check_bare_story_counts matches \b([0-9]{2,})[-\s]stor(?:y|ies)\b, and neither "57 are still untouched by any story" nor "all covered now" presents a number next to the word it looks for. The guard is sound and the claim shape slipped past it.

Scope

• Correct the three claims in CONTRIBUTING.md.
• Consider deriving them instead of writing them, the way docs/action-reference.md is generated. A check_evidence_claims.py rule for "N ... untouched by any story" would keep the number honest, and the derivation above is already the whole implementation.

Difficulty

easy as a documentation fix. medium if you also add the guard rule, which is the version worth doing.

Getting started

CONTRIBUTING.md has the build and test commands. No CLA and no copyright waiver. The project is MIT.

[vladimirrott]

The four wrong figures are now corrected on main in #242, so the numbers a contributor reads are right today. What remains here is the part that keeps them right, which was always the more valuable half.

Corrected:

• CONTRIBUTING.md: "57 untouched, plus 8 Fedora-only" becomes 59 All, 10 Fedora-only, and 5 Ubuntu-only, which the sentence never mentioned
• CONTRIBUTING.md: "The Debian-only ones are all covered now" becomes the five that have none, the four fail2ban actions and GrubSetKargs
• ROADMAP.md: "~59 Observer-level" becomes 63
• docs/testing/user-stories.md: the Ubuntu family "a further 50" becomes 79

Also fixed alongside, since it was the same class: docs/architecture.md listed four CallerRole values in one place and three in two others, dropping Boot from both.

Remaining scope for this issue: derive these rather than restate them. Two concrete gaps in check_bare_story_counts let all of it through, and both are worth closing:

1. It matches \b([0-9]{2,})[-\s]stor(?:y|ies)\b, so "57 are still untouched by any story" presents no number beside the word it looks for.
2. It iterates text.splitlines(), so the wrapped "54\nstories" in user-stories.md never matches even though whole-text matching would find it.

A rule for the uncovered-action count would have caught the first, and switching that scan to whole-text with normalised whitespace catches the second. The derivation is small enough to lift straight out of #233. Worth a mutation test in both directions, since the whole reason this issue exists is a guard that reported success.

[vladimirrott]

Re-derived on main today. The documentation half of this is already done, and the issue is now only about the guard.

Running the script from the body against the current checkout:

uncovered today: {'All': 59, 'Fedora': 10, 'Ubuntu': 5} total 74
Ubuntu-only uncovered: ConfigureFail2banJail, Fail2banBanIp, Fail2banStatus, Fail2banUnbanIp, GrubSetKargs

CONTRIBUTING.md:50 now reads:

The suite is 133 stories: 54 atomic + 79 Ubuntu. Every Debian-only action now has one. What is left is the cross-family middle: of the action names available on both families, 59 are still untouched by any story, plus 10 Fedora-only and 5 Ubuntu-only ones.

59, 10 and 5 all match. 133 matches the file count in tests/e2e/stories/, and the five Ubuntu-only ones are named rather than claimed as covered. #308 fixed the prose.

What is left

The guard, which is the half this issue itself called "the version worth doing".

check_bare_story_counts matches \b([0-9]{2,})[-\s]stor(?:y|ies)\b. "59 are still untouched by any story" puts the number nowhere near the word, so nothing screens it. The three figures are right today because someone recomputed them by hand, which is the state that produced the original discrepancy.

The derivation in the body is the whole implementation. A rule that reads docs/action-reference.md and tests/e2e/stories/*.sh, recomputes the three counts, and compares them against the sentence in CONTRIBUTING.md would keep them honest without anyone remembering to.

Two caveats from the body still apply and belong in the rule's docstring: it counts an action as covered when its name appears as a quoted literal anywhere in a story, which over-counts (ConfigureWifi, GrubSetKargs and SetLocale appear only in "must not confuse with" comments), and covered-by-any-story is not the same as exercised on Ubuntu.

Relabelling from easy to medium, since the documentation edit that made it easy is gone.

[bferanmi806-sketch]

Yep, I’d like to take this one. The connection to #341/#353 makes sense, and I’d be happy to work through the guard and mutation coverage.

Feel free to mark it claimed.

[vladimirrott]

@bferanmi806-sketch before you start, a correction, and it is mine to make.

I filed this on 17 August. #242 landed the next day and fixed two of the three claims in the body. I offered you the issue this afternoon without rereading it, so what you accepted was describing a CONTRIBUTING.md that stopped existing two weeks ago. Open it tonight and you would have gone hunting for a 57 and an 8 that have read 59 and 10 since 18 August.

The body is rewritten. What is actually left at 60af0ad:

• CONTRIBUTING.md:50 still says "Every Debian-only action now has one". Line 47, four lines above it in the same table, says five of them have none. Line 47 is the true one, and all five return zero:

Fail2banStatus: 0
Fail2banBanIp: 0
Fail2banUnbanIp: 0
ConfigureFail2banJail: 0
GrubSetKargs: 0

• Nothing in scripts/check_evidence_claims.py derives any of these figures. check_bare_story_counts matches \b([0-9]{2,})[-\s]stor(?:y|ies)\b, which catches "133 stories" and misses both "59 are still untouched by any story" and "54 atomic".

So the numbers are right, and nothing keeps them right. Somebody corrected them by hand in August and the next story added or retired puts them back out. That is the part worth your evening, and it was always the point of the issue; the two-line documentation fix is incidental.

One way this differs from #341 and #353. There the guard was wrong and the tree was right, so a fixture proving the guard bites was straightforward. Here the numbers agree with the tree today, so a rule that only asserts agreement passes on the current file and would have passed on the wrong one too. The fixture carrying the wrong figure is the test that matters.

Labelled claimed, and it is yours.