feat(autofix): escalate stopped takeover PRs and age out unanswered pauses - #8960
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Takeover PRs the autofix loop stops on (round cap, consecutive-failure or time-budget breaker) now get an
autofix/needs-humanlabel, applied by the scan's cap branch on every cap detection — so the comment dedup no longer hides them, and the already-paused fleet backfills automatically on the regular scan rotation. The label is removed wherever management resumes (re-arm, engage, /retry, first-pickup ack) or a human releases the PR.The Fleet Shepherd now covers the takeover pool: the dashboard gains a "Takeover pool" table (state, stop reason, pause age, stale-first) and a read-only "Awaiting human" section so auto-released PRs stay visible. One bounded lever is added: a PR carrying both labels whose pause went unanswered for
AUTO_RELEASE_DAYS(default 3, tunable via repo variable) gets its takeover label removed plus one bilingual summary;needs-humanstays as the filterable TODO. Resume evidence newer than the pause notice — bot resume markers, a trusted human's/takeoveror/retrycommand, a freshlabeledevent — vetoes the release; every read fails closed; a per-tick cap bounds blast radius.Design doc:
docs/design/takeover-fleet-visibility.md.Why it's needed
As of 2026-08-11, 35 open PRs carry
autofix/takeover, and five of them hit the 100-round cap around 2026-08-06 and sat silently for days — no label, no dashboard entry, nobody re-armed them, three with unresolved merge conflicts. The Fleet Shepherd dashboard only tracked bot-authored PRs (3 today), so the entire human takeover pool was invisible: there was no way to see which PRs were paused, why, or for how long without opening each one. Separately, paused PRs never released their takeover label, so the managed pool filled with zombies.Reviewer Test Plan
How to verify
The executable test surface is the repo's behavioral workflow suites (vitest + verbatim bash/jq extraction replayed against stubbed
gh):Expected: 177 tests pass (161 autofix + 16 shepherd), including the new pins: label write outside the comment dedup gate (bootstrap ordering), six removal sites with 404 tolerance, the ack-job gate excluding base-refused/skip-blocked, pause-detection jq replays (bot-marker provenance, forged-marker rejection), the re-arm guard, the release gate ordering (budget → live recheck → scope), and a cross-file contract test that replays qwen-autofix.yml's terminal headlines against the shepherd's stop-reason regex so drift fails CI instead of the dashboard.
actionlint -oneline -pyflakes= -shellcheck= -ignore 'SC2002:' -ignore 'SC2016:' -ignore 'SC2129:' .github/workflows/qwen-autofix.yml .github/workflows/qwen-fleet-shepherd.ymlexits 0 (the bare invocation hangs on this repo's largest workflow even at HEAD — pre-existing; CI uses these flags).Post-merge live verification (workflow changes only run from the default branch): within a few hours the five currently-paused takeover PRs (#8213, #8396, #8416, #8439, #8443) should carry
autofix/needs-human(idle backoff, not minutes); the next shepherd tick should render the takeover table on the dashboard issue;gh workflow run qwen-fleet-shepherd.yml -f dry_run=trueshould list those five as release candidates without writing.Evidence (Before & After)
N/A (CI/workflow change, not user-visible UI). Before: paused takeover PRs invisible, no labels, dashboard shows 3 bot PRs only. After: dashboard takeover table + needs-human labels + bounded auto-release, all replayed in the behavioral suites.
Tested on
(Workflows execute on ubuntu runners in CI; GNU date/jq semantics replayed locally on macOS.)
Environment (optional)
N/A — behavioral test suites + actionlint only.
Risk & Scope
autofix/skipat every gate, caps releases at 3 per tick, and is fully dry-runnable. Worst-case recovery is one/takeovercomment.Linked Issues
None — proactive hardening of the takeover loop based on a 2026-08-11 fleet analysis.
中文说明
本 PR 做了什么
当 autofix 循环在某个 takeover PR 上停止(达到轮次上限、连续失败或超时熔断)时,扫描的 cap 分支现在会在每次检测到触顶时打上
autofix/needs-human标签——评论去重不再掩盖这些 PR,存量已暂停的 PR 也会在常规扫描轮换中自动补标。该标签在管理恢复(re-arm、接管、/retry、首次接管 ack)或人工释放时被移除。Fleet Shepherd 现在覆盖 takeover 池:dashboard 新增 "Takeover pool" 表(状态、停止原因、暂停时长,按陈旧度优先)和只读的 "Awaiting human" 段,保证被自动释放的 PR 仍然可见。新增一个有界杠杆:同时携带两个标签且暂停超过
AUTO_RELEASE_DAYS天(默认 3,可通过仓库变量调整)的 PR 会被移除 takeover 标签并收到一条双语总结;needs-human保留为可过滤的待办。比暂停通知更新的恢复证据——bot 的恢复标记、可信用户的/takeover或/retry命令、新的labeled事件——会否决释放;所有读取失败即关闭(fail-closed);每 tick 上限控制影响面。设计文档:
docs/design/takeover-fleet-visibility.md。为什么需要
截至 2026-08-11,35 个 open PR 携带
autofix/takeover,其中 5 个在 2026-08-06 前后打到 100 轮上限后静默了多天——无标签、无 dashboard 记录、无人重新武装,其中 3 个还带着未解决的合并冲突。Fleet Shepherd 的 dashboard 此前只跟踪 bot 自己的 PR(今天 3 个),整个人类 takeover 池完全不可见:不逐个打开 PR 就无法知道哪些被暂停、为何暂停、停了多久。此外,暂停的 PR 从不释放 takeover 标签,导致托管池被"僵尸"占满。评审者测试计划
可执行测试面是仓库既有的行为级 workflow 测试套件(vitest + 对 stub
gh逐字回放 bash/jq):预期:177 个测试全部通过(161 autofix + 16 shepherd),包含新增的钉:标签写入在评论去重门之外(bootstrap 顺序)、6 处 404 容忍的移除点、ack job 排除 base-refused/skip-blocked 的门、暂停检测 jq 回放(bot 标记来源校验、伪造标记拒绝)、re-arm 守卫、释放门顺序(预算 → live 复核 → scope),以及一个跨文件契约测试——把 qwen-autofix.yml 的终态 headline 逐一对上 shepherd 的停止原因正则,漂移会在 CI 失败而不是 dashboard 静默退化。
actionlint -oneline -pyflakes= -shellcheck= -ignore 'SC2002:' -ignore 'SC2016:' -ignore 'SC2129:' .github/workflows/qwen-autofix.yml .github/workflows/qwen-fleet-shepherd.yml退出码为 0(裸 actionlint 在本仓库最大的 workflow 上即使 HEAD 也会挂起——既有问题,CI 使用上述参数)。合并后现网验证(workflow 改动只能从默认分支运行):几小时内 5 个当前暂停的 takeover PR(#8213、#8396、#8416、#8439、#8443)应携带
autofix/needs-human(受 idle backoff 影响,非分钟级);下一个 shepherd tick 应在 dashboard issue 上渲染 takeover 表;gh workflow run qwen-fleet-shepherd.yml -f dry_run=true应列出这 5 个 PR 为释放候选且不产生写入。证据(前后对比)
N/A(CI/workflow 改动,无用户可见 UI)。改动前:暂停的 takeover PR 不可见、无标签、dashboard 只有 3 个 bot PR;改动后:dashboard takeover 表 + needs-human 标签 + 有界自动释放,全部在行为测试套件中回放验证。
测试平台
macOS ✅ 已测;Windows / Linux N/A(workflow 在 CI 的 ubuntu runner 执行;GNU date/jq 语义已在 macOS 本地回放)。
风险与范围
autofix/skip,每 tick 限 3 个,支持 dry-run。最坏情况的恢复只需一句/takeover评论。