Canary 403 NOT fixed by #745/#746 — release-manager App can't bypass tag ruleset even repo-scoped; need effective-permission diagnostic

[don-petry]

The #745 hypothesis was wrong — token scoping is NOT the cause

PR #746 (repo-scoped CANARY_WRITE_TOKEN) merged and is live, but the first promotion-due run after it (#29489643941, 10:06Z) still failed all 4 moves with the identical error:

_gh_move_tag: could not move refs/tags/agent-shield/v2-ring0 (HTTP):
{"message":"Resource not accessible by integration","status":"403"}

Verified in that run:

• The write token is minted and used: _gh_write routes PATCH /git/refs/tags through CANARY_WRITE_TOKEN (canary-rollout.sh:120-122, 144).
• It is repo-scoped correctly: repositories: [.github, .github-private], permission-contents: write.
• Still 403. So an owner-wide token AND a repo-scoped token both fail — token scoping was never the cause.

What is definitively established

• OrgAdmin can move the tag — a manual admin-token PATCH (no-op, same sha) of auto-rebase/v2-ring0 returned 200. API + ruleset + tag state are healthy.
• The release-manager App (4193127) cannot — 403 on the same endpoint, regardless of token scope.
• Nothing config-side changed across the break: release-channel-tags ruleset (#18516227) last updated 2026-07-04 (bypass_actors = OrgAdmin + 3167543 + 4193127, all mode=always); App installation last updated 2026-07-11. Last successful move was 2026-07-13 20:55Z (#29284341779); first failure 2026-07-15 06:12Z.
• "Resource not accessible by integration" is a permission/accessibility 403 — NOT a "repository rule violations" message — which is odd given the App both has contents:write AND is a listed bypass actor.

Ask: get DATA, don't guess again

1. Add a one-shot diagnostic in the _gh_move_tag failure path (behind the existing un-suppress from feat: implement issue #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #744): on a 403, dump the write-token's effective permissions — e.g. gh api -i /repos/petry-projects/.github and print X-Accepted-GitHub-Permissions / the /installation permissions the token actually carries, plus whether the token is repo- or owner-scoped. This distinguishes:
   • (a) token lacks effective contents:write → minting bug (code-fixable — e.g. the permission-contents: write request isn't being honored owner-side, or the App's install perm on the specific repo differs from the org view).
   • (b) token HAS contents:write but the ruleset bypass isn't applied → NOT a code bug. A maintainer must re-save the release-channel-tags bypass_actors (re-applying sometimes re-establishes an Integration bypass that silently lapsed) or open GitHub support. Do not loosen/remove the ruleset.
2. Post the diagnostic output on this issue so the next step is decided from data.

Guardrails

• No change to immutable vX.Y.Z release tags or to the refs/tags/** ruleset rules without maintainer sign-off.
• Keep feat: implement issue #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #744's un-suppressed error and feat: implement issue #745 — Canary tag-move 403 'Resource not accessible by integration' — release-manager App token can't PATCH protected channel tags (follow-up to #743/#744) #746's two-token structure (both are correct and useful regardless of this root cause).

Refs

• Prior: Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #743→feat: implement issue #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #744 (diagnostic, merged), Canary tag-move 403 'Resource not accessible by integration' — release-manager App token can't PATCH protected channel tags (follow-up to #743/#744) #745→feat: implement issue #745 — Canary tag-move 403 'Resource not accessible by integration' — release-manager App token can't PATCH protected channel tags (follow-up to #743/#744) #746 (token scoping, merged, did NOT resolve).
• Failing runs incl. post-fix: #29489643941 (10:06Z). Last good: #29284341779 (07-13 20:55Z).

[don-petry]

Dev-Lead Implementation Plan

Issue: #749 — Canary 403 NOT fixed by #745/#746 — release-manager App can't bypass tag ruleset even repo-scoped; need effective-permission diagnostic

Scope

Add a one-shot, data-gathering effective-permission diagnostic to the _gh_move_tag 403 failure path (behind #744's un-suppressed error) that dumps the write token's X-Accepted-GitHub-Permissions header and installation scope — distinguishing a minting bug (token lacks effective contents:write) from a lapsed ruleset bypass (token has it but the PATCH is still blocked). No ruleset/tag changes, no behavior change to the happy path.

Implementation Checklist

• scripts/canary-rollout.sh: add _gh_403_diag <repo> helper — introspects via the SAME write token (_gh_write): captures X-Accepted-GitHub-Permissions from gh api -i repos/<repo>, and repository_selection/total_count from gh api /installation/repositories to classify OWNER-WIDE vs REPO-SCOPED. Best-effort, always returns 0.
• scripts/canary-rollout.sh: wire _gh_403_diag into the _gh_move_tag non-404 PATCH failure branch, gated on a 403 / "not accessible by integration" signal, emitted before the existing ::error::. POST create-fallback and 422 behavior (Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #743) unchanged.

Tests to Write (tests/canary_rollout.bats)

• 403 PATCH failure → diagnostic dumps X-Accepted-GitHub-Permissions + scope, still surfaces ::error::, still does NOT fall back to POST
• Diagnostic introspection routes through CANARY_WRITE_TOKEN
• repository_selection=all → reports OWNER-WIDE; selected → REPO-SCOPED
• A non-403 (422 ruleset) failure does NOT trigger the diagnostic (preserves Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #743)

Test Command

npx bats tests/canary_rollout.bats (full: bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck + frontmatter)

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• scripts/canary-rollout.sh: added _gh_403_diag <repo> — introspects with the SAME write token the PATCH used (_gh_write → CANARY_WRITE_TOKEN when set); dumps X-Accepted-GitHub-Permissions from gh api -i repos/<repo> and the installation repository_selection/total_count, classifying OWNER-WIDE (all) vs REPO-SCOPED (selected). Best-effort, always returns 0.
• scripts/canary-rollout.sh: wired it into the _gh_move_tag non-404 PATCH-failure branch, gated on a 403 signal (*403* or *not accessible by integration*), emitted before feat: implement issue #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #744's ::error::. POST create-fallback and the Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #743 422 behavior are unchanged.
• tests/canary_rollout.bats: 4 new tests (403 dumps header+scope & still surfaces error & no POST fallback; introspection uses CANARY_WRITE_TOKEN; all→OWNER-WIDE; 422 does NOT trigger the diagnostic).

What this does NOT do (per guardrails)

• No change to the release-channel-tags ruleset, to refs/tags/** rules, or to any immutable vX.Y.Z tag.
• Keeps feat: implement issue #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #744's un-suppressed error and feat: implement issue #745 — Canary tag-move 403 'Resource not accessible by integration' — release-manager App token can't PATCH protected channel tags (follow-up to #743/#744) #746's two-token structure intact.

How to read the next failing run

On the next promotion-due 403, the _gh_move_tag 403 effective-permission diagnostic (#749) group in the run log decides the root cause from data:

• X-Accepted-GitHub-Permissions shows contents=write AND scope is REPO-SCOPED → the token carries the perm → case (b): the ruleset bypass lapsed. A maintainer must re-save release-channel-tags bypass_actors (or open GitHub support). Not a code bug — do not loosen the ruleset.
• The header lacks contents=write / the repo read itself 403s → case (a): a minting bug (the permission-contents: write request isn't honored on this repo). Code-fixable.

Test Results

$ npx bats tests/canary_rollout.bats
ok 116 _gh_move_tag: on a 403 dumps the effective-permission diagnostic then surfaces the error (#749)
ok 117 _gh_move_tag: the 403 diagnostic introspects through CANARY_WRITE_TOKEN (#749)
ok 118 _gh_move_tag: the 403 diagnostic reports OWNER-WIDE when repository_selection is all (#749)
ok 119 _gh_move_tag: a non-403 (422 ruleset) failure does NOT trigger the 403 diagnostic (#749)
...
231 tests, 0 failures

$ npx bats -r test tests    → 838 tests, 0 failures
$ bash .dev-lead/scripts/dev-lead-lint.sh   → all checks passed (shellcheck 23 files + frontmatter)
$ shellcheck -S warning scripts/canary-rollout.sh   → clean

Files Changed

• scripts/canary-rollout.sh: new _gh_403_diag helper + 403-gated call in _gh_move_tag.
• tests/canary_rollout.bats: 4 new tests + _diag_403_stub.

Notes

The diagnostic is a pure read (never mutates, never re-issues the PATCH) and prints only permission/scope classification — never the token value. It runs at most once per failing move, only on a 403. Next step: read the diagnostic output from the first post-merge promotion-due 403 run and post it here so case (a) vs (b) is decided from data.