Skip to content

feat(autofix): escalate stopped takeover PRs and age out unanswered pauses - #8960

Merged
wenshao merged 20 commits into
mainfrom
feat/takeover-fleet-visibility
Aug 15, 2026
Merged

feat(autofix): escalate stopped takeover PRs and age out unanswered pauses#8960
wenshao merged 20 commits into
mainfrom
feat/takeover-fleet-visibility

Conversation

@wenshao

@wenshao wenshao commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Takeover PRs the autofix loop stops on (round cap, consecutive-failure or time-budget breaker) now get an autofix/needs-human label, applied by the scan's cap branch on every cap detection — so the comment dedup no longer hides them, and the already-paused fleet backfills automatically on the regular scan rotation. The label is removed wherever management resumes (re-arm, engage, /retry, first-pickup ack) or a human releases the PR.

The Fleet Shepherd now covers the takeover pool: the dashboard gains a "Takeover pool" table (state, stop reason, pause age, stale-first) and a read-only "Awaiting human" section so auto-released PRs stay visible. One bounded lever is added: a PR carrying both labels whose pause went unanswered for AUTO_RELEASE_DAYS (default 3, tunable via repo variable) gets its takeover label removed plus one bilingual summary; needs-human stays as the filterable TODO. Resume evidence newer than the pause notice — bot resume markers, a trusted human's /takeover or /retry command, a fresh labeled event — vetoes the release; every read fails closed; a per-tick cap bounds blast radius.

Design doc: docs/design/takeover-fleet-visibility.md.

Why it's needed

As of 2026-08-11, 35 open PRs carry autofix/takeover, and five of them hit the 100-round cap around 2026-08-06 and sat silently for days — no label, no dashboard entry, nobody re-armed them, three with unresolved merge conflicts. The Fleet Shepherd dashboard only tracked bot-authored PRs (3 today), so the entire human takeover pool was invisible: there was no way to see which PRs were paused, why, or for how long without opening each one. Separately, paused PRs never released their takeover label, so the managed pool filled with zombies.

Reviewer Test Plan

How to verify

The executable test surface is the repo's behavioral workflow suites (vitest + verbatim bash/jq extraction replayed against stubbed gh):

npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/qwen-autofix-workflow.test.js scripts/tests/qwen-fleet-shepherd-workflow.test.js

Expected: 177 tests pass (161 autofix + 16 shepherd), including the new pins: label write outside the comment dedup gate (bootstrap ordering), six removal sites with 404 tolerance, the ack-job gate excluding base-refused/skip-blocked, pause-detection jq replays (bot-marker provenance, forged-marker rejection), the re-arm guard, the release gate ordering (budget → live recheck → scope), and a cross-file contract test that replays qwen-autofix.yml's terminal headlines against the shepherd's stop-reason regex so drift fails CI instead of the dashboard.

actionlint -oneline -pyflakes= -shellcheck= -ignore 'SC2002:' -ignore 'SC2016:' -ignore 'SC2129:' .github/workflows/qwen-autofix.yml .github/workflows/qwen-fleet-shepherd.yml exits 0 (the bare invocation hangs on this repo's largest workflow even at HEAD — pre-existing; CI uses these flags).

Post-merge live verification (workflow changes only run from the default branch): within a few hours the five currently-paused takeover PRs (#8213, #8396, #8416, #8439, #8443) should carry autofix/needs-human (idle backoff, not minutes); the next shepherd tick should render the takeover table on the dashboard issue; gh workflow run qwen-fleet-shepherd.yml -f dry_run=true should list those five as release candidates without writing.

Evidence (Before & After)

N/A (CI/workflow change, not user-visible UI). Before: paused takeover PRs invisible, no labels, dashboard shows 3 bot PRs only. After: dashboard takeover table + needs-human labels + bounded auto-release, all replayed in the behavioral suites.

Tested on

OS Status
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

(Workflows execute on ubuntu runners in CI; GNU date/jq semantics replayed locally on macOS.)

Environment (optional)

N/A — behavioral test suites + actionlint only.

Risk & Scope

  • Main risk or tradeoff: the auto-release lever removes a takeover label — a wrong release would silently drop a PR from management. Mitigations: the lever requires both labels AND a pause older than 3 days AND no newer resume evidence (bot markers, trusted commands, labeled events), re-reads live labels fail-closed immediately before the write, honors autofix/skip at every gate, caps releases at 3 per tick, and is fully dry-runnable. Worst-case recovery is one /takeover comment.
  • Not validated / out of scope: the shepherd's other levers (conflict dispatch, stale-base sync) stay scoped to bot-authored PRs; no changes to round caps, breakers, or review-bot behavior; post-merge live behavior (Group D in the test plan) is verified after merge because workflow_dispatch only runs default-branch workflows.
  • Breaking changes / migration notes: none. The new label is created idempotently with a fixed color on first use.

Linked Issues

None — proactive hardening of the takeover loop based on a 2026-08-11 fleet analysis.

中文说明

本 PR 做了什么

当 autofix 循环在某个 takeover PR 上停止(达到轮次上限、连续失败或超时熔断)时,扫描的 cap 分支现在会在每次检测到触顶时打上 autofix/needs-human 标签——评论去重不再掩盖这些 PR,存量已暂停的 PR 也会在常规扫描轮换中自动补标。该标签在管理恢复(re-arm、接管、/retry、首次接管 ack)或人工释放时被移除。

Fleet Shepherd 现在覆盖 takeover 池:dashboard 新增 "Takeover pool" 表(状态、停止原因、暂停时长,按陈旧度优先)和只读的 "Awaiting human" 段,保证被自动释放的 PR 仍然可见。新增一个有界杠杆:同时携带两个标签且暂停超过 AUTO_RELEASE_DAYS 天(默认 3,可通过仓库变量调整)的 PR 会被移除 takeover 标签并收到一条双语总结;needs-human 保留为可过滤的待办。比暂停通知更新的恢复证据——bot 的恢复标记、可信用户的 /takeover/retry 命令、新的 labeled 事件——会否决释放;所有读取失败即关闭(fail-closed);每 tick 上限控制影响面。

设计文档:docs/design/takeover-fleet-visibility.md

为什么需要

截至 2026-08-11,35 个 open PR 携带 autofix/takeover,其中 5 个在 2026-08-06 前后打到 100 轮上限后静默了多天——无标签、无 dashboard 记录、无人重新武装,其中 3 个还带着未解决的合并冲突。Fleet Shepherd 的 dashboard 此前只跟踪 bot 自己的 PR(今天 3 个),整个人类 takeover 池完全不可见:不逐个打开 PR 就无法知道哪些被暂停、为何暂停、停了多久。此外,暂停的 PR 从不释放 takeover 标签,导致托管池被"僵尸"占满。

评审者测试计划

可执行测试面是仓库既有的行为级 workflow 测试套件(vitest + 对 stub gh 逐字回放 bash/jq):

npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/qwen-autofix-workflow.test.js scripts/tests/qwen-fleet-shepherd-workflow.test.js

预期:177 个测试全部通过(161 autofix + 16 shepherd),包含新增的钉:标签写入在评论去重门之外(bootstrap 顺序)、6 处 404 容忍的移除点、ack job 排除 base-refused/skip-blocked 的门、暂停检测 jq 回放(bot 标记来源校验、伪造标记拒绝)、re-arm 守卫、释放门顺序(预算 → live 复核 → scope),以及一个跨文件契约测试——把 qwen-autofix.yml 的终态 headline 逐一对上 shepherd 的停止原因正则,漂移会在 CI 失败而不是 dashboard 静默退化。

actionlint -oneline -pyflakes= -shellcheck= -ignore 'SC2002:' -ignore 'SC2016:' -ignore 'SC2129:' .github/workflows/qwen-autofix.yml .github/workflows/qwen-fleet-shepherd.yml 退出码为 0(裸 actionlint 在本仓库最大的 workflow 上即使 HEAD 也会挂起——既有问题,CI 使用上述参数)。

合并后现网验证(workflow 改动只能从默认分支运行):几小时内 5 个当前暂停的 takeover PR(#8213#8396#8416#8439#8443)应携带 autofix/needs-human(受 idle backoff 影响,非分钟级);下一个 shepherd tick 应在 dashboard issue 上渲染 takeover 表;gh workflow run qwen-fleet-shepherd.yml -f dry_run=true 应列出这 5 个 PR 为释放候选且不产生写入。

证据(前后对比)

N/A(CI/workflow 改动,无用户可见 UI)。改动前:暂停的 takeover PR 不可见、无标签、dashboard 只有 3 个 bot PR;改动后:dashboard takeover 表 + needs-human 标签 + 有界自动释放,全部在行为测试套件中回放验证。

测试平台

macOS ✅ 已测;Windows / Linux N/A(workflow 在 CI 的 ubuntu runner 执行;GNU date/jq 语义已在 macOS 本地回放)。

风险与范围

  • 主要风险/权衡:自动释放杠杆会移除 takeover 标签——错误释放会让 PR 静默脱离管理。缓解:要求双标签 + 暂停超过 3 天 + 无更新的恢复证据(bot 标记、可信命令、labeled 事件),写入前即时 fail-closed 复核标签,每个门都尊重 autofix/skip,每 tick 限 3 个,支持 dry-run。最坏情况的恢复只需一句 /takeover 评论。
  • 未验证/范围外:shepherd 的其他杠杆(冲突分发、stale-base 同步)仍只作用于 bot 的 PR;不改动轮次上限、熔断器或 review bot 行为;合并后的现网行为(测试计划 Group D)在合并后验证,因为 workflow_dispatch 只运行默认分支上的 workflow。
  • 破坏性变更/迁移说明:无。新标签首次使用时以固定颜色幂等创建。

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants