fix(generate): repair lossy regenerations and hold back the rules that still fail

[dawsontoth]

One rule whose regeneration fails its checks no longer fails the whole docs auto-sync. The generator now runs the validators’ checks on each regenerated rule and asks the model to fix exactly what is wrong, up to two times. Any rule that still fails is held back at its committed version, so the sync PR opens with everything else and names the held-back rules. The sync PR body is also rebuilt from the branch on every run, so it lists every rule the PR changes, not only the latest run's, and repairs are recorded in the sync commits. Failures go to one stable issue instead of one issue per docs SHA, and the three chronically lossy rules get must_cover pins. Closes #118.

❓ Your call: post-review delta, not re-reviewed. It has two parts. 627b190 adds 6 lines and removes 2 in sync-report.mjs: a fallback to "no recorded baseline" when --base is not where the branch diverged, from the last round's nits. e6e066d trims comments only, from Ethan's review.

For the human reviewer

1. Planning review: better-alternative-exists, adopted. The review's alternative: when a run has nothing new to commit but an open sync PR exists, refresh that PR's body instead of failing the run. That became the no-diff refresh, widened in entry 9. The review's other findings were adopted too:

   • The generator now applies the structural checks (H1, minimum length, leaked MDX).
   • A rule is held back only when its committed body passes the same checks.
   • The HEAD baseline is strict: only a missing path counts as "no baseline".
   • A failed issue lookup is logged.
   • The docs ref reaches the shell through env.

   Overruled, with the deciding fact for each:

   • TypeScript for the new modules. Every script under scripts/ is .mjs, run directly by node from package.json and the workflow.
   • Retrying with a fresh call (plain, or with an augmented must_cover) instead of a conversation. The 27 failed runs were 27 fresh samples, and querying-rest-apis failed every one. It lost at least 18 distinct facts at about 9 per run, a different subset each time, so a fresh retry trades one set of drops for another. A repair edits the draft that already kept the rest. Nothing measures the real model's repair success yet (see Verification).
   • Not opening an issue when the lookup fails. Skipping would leave the failure with no trace, which is what the 2026-09-07 run left. A duplicate issue is one click to close.

2. The requirement, as filed. Auto-sync has failed on every docs deploy since 2026-09-14: regeneration keeps dropping facts #118 asks for four things: repair, isolation, pins for the chronic rules, and a single failure issue. Its "Done when" is met by those four, and all four are implemented except one sub-item: querying-rest-apis is not split. Splitting is a taxonomy change that needs a real-model regeneration to judge, and the pins plus repair target the drops measured in CI. Entry 9's PR-provenance fix began as a separate follow-up task. It is folded in here because it replaces this PR's earlier held-back splice, and because this repo runs CI only on PRs into main. No other open PR touches the pipeline, and no auto/docs-sync PR is open, so the first run after merge starts clean from main.

3. Held back with nothing to commit and no open sync PR fails the run, so the failure issue names the rules. The alternative was a green no-op run, which would let a chronically lossy rule stay stale without anyone being told. The fix is cheap to reverse. Cost of "no": held-back rules would show only in the job log.

4. A model error still stops the run. This applies to API, auth, and max_tokens truncation errors alike, and is the same as before except that the report now names the rule. Holding back on these was the first draft. The review showed the problem: an expired key would hold back every rule, and with an open sync PR the run would refresh its body and stay green with nothing synced. Only check failures are held back. Truncation is new as an error: before, a cut-off body was written like any other.

5. The repair budget defaults to 2 (GENERATE_MAX_REPAIRS), so a rule costs at most 3 model calls. Each repair continues the same conversation and gets the whole body back.

6. 17 must_cover anchors added; 3 of them replace the bare select(/sort(/limit(. Each was dropped in at least 7 of the 27 failed runs, and each is in both the current docs and the committed body. instanceof, logger and import() are generic tokens. But each is exactly the fact the retention check enforces, and each occurs once in its source (import() twice), so a passing mention is the fact. If the docs later drop one, that rule is held back and named instead of failing every sync.

7. The failure issue is never closed automatically. A successful sync leaves it open for a human to close, as before.

8. Two stops kept on purpose.

   • Any model error stops the run, including a 529 that outlasts the SDK's retries. That blocks every rule, as it did before this PR, but an auth failure can no longer go green.
   • A hold-back is refused when the rule's frontmatter no longer reconciles with the manifest. That state can only arise if a manifest edit lands without its regenerated rule, which the PR-time validate-generated check already refuses.

9. The sync PR body is rebuilt from the branch on every run. Before, it was composed from a snapshot of what was stale before the latest run. So on the rolling branch it listed only that run's rules, and every update replaced the list: given run 2's snapshot, the old code lists only b when the PR also changes a.

   • It now lists every rule file the branch changes against main, with main's recorded baseline for each. Human review commits on the branch are included, and so are added and deleted files.
   • Repairs are recorded as Sync-Repaired: trailers on the sync commit, written before the push, and read back from the branch. The latest commit to touch a rule decides. An earlier draft kept them in a hidden comment in the PR body. The review showed that a run which pushed and then failed to update the body lost them for good.
   • The whole body is overwritten on every run, including runs with nothing new to commit, so hand edits to it don't survive. Before, they were already overwritten on every run that committed.
   • A merge commit does not clear a rule's repair record. The record is advisory, and a merge that makes the rule match main drops it from the list anyway.

Changes

• lib/retention.mjs (new): the fact-retention definition, moved out of validate-generated.mjs so the generator and the validator cannot disagree. The semantics are the same, with one exception: bodyAtHead now returns null only for a path absent from a real HEAD commit, and throws on any other git failure (an unborn HEAD included) with git's own message. anchorsBeyondFacts lets a backtick-wrapped anchor and the bare fact it pins count once.
• lib/body-checks.mjs (new): the H1 / minimum-length / leaked-MDX checks, shared with validate-generated.mjs. The H1 condition mirrors validate-skills.mjs.
• lib/regenerate.mjs (new): checkBody (every check the validators apply to a generated body), the repair loop regenerateFaithfully, and parseMaxRepairs.
• lib/llm.mjs: generateRuleBody becomes ruleConversation, with generate() and repair() over one message history and an injectable createMessage for tests. The repair prompt quotes the committed line each dropped fact came from, asks for an anchor only once when it is also a dropped fact, and lists structural problems. A max_tokens stop is now an error.
• generate-rules.mjs:
  • Runs the loop with the HEAD body as the baseline (the same one the validator uses).
  • Holds a rule back by skipping its write before AGENTS.md is assembled, but only if the existing file passes everything validate-generated checks: checkBody and frontmatter reconciliation. Otherwise, or for a rule with no file, or on a model error, the run stops.
  • Stops are StopRun throws. main() writes --report with the rule and reason it stopped on. A report that cannot be written fails a run that otherwise succeeded.
  • The loop moved into run(), unindented.
  • Adds --report and GENERATE_MAX_REPAIRS.
• lib/render.mjs: frontmatterProblems, the frontmatter reconciliation moved out of validate-generated.mjs with identical messages.
• lib/generation-report.mjs (new):
  • renders the "Held back" section, the "Repaired" section, and the failure details for the failure issue;
  • writes and parses Sync-Repaired: trailers.
• sync-report.mjs:
  • --format pr-body now derives the changed rules and the repairs from the branch against --base;
  • adds --format failure-details and --format commit-trailers;
  • drops --from and the pre-sync snapshot it read.
• validate-generated.mjs: imports the shared definitions (retention, body checks, frontmatter reconciliation); behavior is unchanged apart from the stricter baseline read.
• generate.yaml:
  • drops the pre-sync snapshot step;
  • passes --report, and tees the generator and Validate output;
  • rebuilds the PR body on every run;
  • with nothing new to commit, it refreshes an open sync PR's body, or fails when there is no open PR and something was held back;
  • writes the run's repairs into the sync commit;
  • moves the failure step to the end of the job, gives it the stable title and a paginated REST lookup, and has it quote the stop and the held-back rules plus the failed step's log in a REST comment;
  • passes docs_ref and payload refs to the shell through env.
• rules.manifest.yaml: pins for three rules.
  • querying-rest-apis: function signatures replace the bare select(/sort(/limit(, plus one anchor per type-conversion row.
  • v5-upgrade: instanceof, import(), logger.
  • automatic-apis: Sec-WebSocket-Protocol: mqtt, @hidden, static hidden = true.
• CONTRIBUTING.MD: documents the checks, repair and hold-back, how the PR body is rebuilt, and the stable issue, what to do about a held-back or repaired rule, and who owns what. It also drops the pre-sync snapshot step from the pipeline description.

Verification

Route (c): a live run through the production entry points, with the model replaced at the HTTP boundary. A local fake /v1/messages server (ANTHROPIC_BASE_URL) replays each committed rule body with facts removed:

• querying-rest-apis: drops the name==/sort( lines, on the first call only.
• automatic-apis: rewords Sec-WebSocket-Protocol: mqtt, on the first call only.
• v5-upgrade: drops instanceof, on every call.

The docs were a fresh build of HarperFast/documentation@2cfb813 (docs main, 2026-10-06).

• Base origin/main (de2c13a): generate-rules.mjs writes all 9 stale rules. validate-generated.mjs --docs-path then fails with CI's signature: 4 errors, including querying-rest-apis: regeneration dropped 10 facts still documented in its sources. The whole sync is refused.
• This branch, run 1:
  • automatic-apis is repaired in 1 pass.
  • querying-rest-apis is repaired in 1 pass, restoring 10 facts.
  • v5-upgrade is held back after 2 repairs.
  • Result: 8 regenerated and 1 held back. validate-generated --docs-path and npm run validate pass.
• This branch, run 2, on top of run 1's commit: only v5-upgrade is retried (held back again), 21 rules are unchanged, and there is no diff.
• Ineligible fallback: I added an anchor (Prefer: count=exact) that the committed querying-rest-apis body lacks. The run exits 1, reporting that the existing body cannot be kept instead (missing must_cover "Prefer: count=exact"), and nothing is written.
• Old PR-body composer: given run 2's pre-generation snapshot (a already fresh, b stale), the old sync-report --format pr-body --from lists only b.
• Workflow bash: the changed run: blocks, extracted and run against a stub gh:
  • No diff, open PR: the rebuilt body is posted via gh pr edit --body-file, and a second identical run makes no edit.
  • No diff and no open PR: exits 1 with the ::error::.
  • PR lookup fails: exits 1.
  • Commit path, against a scratch bare remote: the sync commit carries the Sync-Repaired: trailer, it pushes, and gh pr create --body-file gets a body listing the changed rule and its repair.
  • Failure issue already open: one REST comment carrying the details and the log tail.
  • No failure issue: creates it and sets the type.
  • Issue lookup fails: a warning, then a new issue.
  • Generator stops on a rule: the comment quotes its ✗ line.
• Not exercised: how well the real model repairs. No API key was available locally, so the first CI run after merge is the first real repair. A manual workflow_dispatch run right after merge would show it.
• generate-rules.test.mjs (new): an end-to-end test of the real generate-rules.mjs in a scratch git repository, against a local Messages API stand-in.
  • First case: one rule stays lossy through 2 repairs, and another is repaired. The lossy rule's file is byte-identical afterwards, the repaired one carries the new sourceCommit, the report lists each, and validate-generated --docs-path passes on the result.
  • Second case: an HTTP 400 on the second rule exits 1, and the report still records the first rule as held back and stoppedOn the second.
  • Third case: a manifest whose description moved on without the rule file stops the run with cannot be kept instead: … "description" diverges, and holds nothing back.
  • Fourth case: an unwritable --report path fails a run that otherwise succeeded.
• sync-report.test.mjs (new): --format pr-body on a scratch sync branch with two runs of commits.
  • Both runs' rules are listed with main's baselines, and the docs range starts at the oldest baseline.
  • The repairs from both runs' trailers are listed; a third, clean regeneration of a rule drops its record.
  • Added and deleted rule files are labelled as such.
• npm test: 86 pass.
  • retention.test.mjs: legacy semantics, plus the strict baseline in scratch repositories: the committed body and not the working tree, null only for a missing path, and a throw with git's message outside a repository or before the first commit.
  • body-checks.test.mjs: structural checks.
  • regenerate.test.mjs: the repair loop's accept, repair, hold-back, new-drop, anchor, structural, waiver, error, and budget paths.
  • llm.test.mjs: the prompts, the conversation's message history, and the truncation/empty errors.
  • generation-report.test.mjs: rendering, failure details, and Sync-Repaired: trailers.
• All 17 added anchors were checked against the current docs sources and the committed bodies; each is present in both.

Related PRs: #119 independent, #98 independent, #26 independent
Complexity: medium

_{Review-Coverage: authored=claude; ran=gemini,cursor-muse,codex; adjudicated=domain; blocked=cursor-grok(no-receipt),claude(fallback)(out-of-budget),cursor-composer(out-of-budget); declined=cursor-kimi; rounds=5; full=1 @ e6e066d}

_{Review-Attention: study ~11m (decisions: repair-record-in-commit-trailers, later-touch-clears-repair, provenance-from-git-diff, stop-on-any-model-error, stop-on-frontmatter-drift, single-rolling-failure-issue, green-refresh-when-held-back-with-open-pr, conversational-repair, do-less-alternative) @ e6e066d}

[gemini-code-assist]

Code Review

This pull request introduces a robust "repair-then-hold-back" loop for automated rule generation, ensuring that lossy or structurally invalid rule bodies do not block the entire documentation sync process. It extracts core validation logic (such as fact retention, structural checks, and frontmatter reconciliation) into reusable library modules (retention.mjs, body-checks.mjs, regenerate.mjs, and generation-report.mjs) shared between the generator and the validator. Additionally, it updates the sync report to detail held-back or repaired rules in PR bodies and failure issues, backed by comprehensive unit and end-to-end tests. Feedback suggests improving Git baseline error messages by appending err.stderr to the thrown errors when Git commands fail.

[Ethan-Arrowood]

LGTM - dispatch review is complaining about some repo inconsistencies but I disagree with them (like it claims everything should be TS even though we have plenty of existing ESM JS). It found no correctness issues and considering the size of this change and flexibility of this package, I'm comfortable with you merging and us fixing forward anything else that is out of place.

_{🤖 Reviewed with Codex}

[github-actions]

🎉 This PR is included in version 1.14.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀