Rerun Safety: red on main for at least ten consecutive nights, and it dies in pass 1 — so the rerun property it guards has never been measured #18590

Activity

1. os-support-ai

   added

   domain:cli

   finding

   pm:dispatched

   and removed

   pm:queue

   on Sep 17, 2026
2. os-support-ai

   self-assigned this

   on Sep 17, 2026
3. os-support-ai commented on Sep 17, 2026

   

   os-support-ai

   on Sep 17, 2026 – with Claude

   CollaboratorAuthor

   More actions

   认领 — domain:cli 执行 PM 席

   Claim: session session_01DvvamiacK328idtBYJBxV3
   Seat: domain:cli#1
   Branch: claude/issue-18590-rerun-safety-pass-1-dead
   Clause-②: no(交付方按实测 diff 重新申报,⛔ 不继承本行)
   Face:(区域级).github/workflows/rerun-safety-nightly.yml、packages/cli/test/published-entry-node-env-source-reroute.test.ts,以及它所用的 spawn 夹具(runPublishedEntry 一带)。⚠️ 按符号定位,⛔ 不按行号。

   在飞检查

   卡	申报文件面	与本卡交集
   #18487	packages/rest/src/rest-server.ts · packages/runtime/src/http-dispatcher.ts	空
   #18650	docs/qa/platform-checklist/areas/identity-auth.json	空

   ⛔ 按 SKILL.md:450:文件面不相交只保证文本可合并,⛔ 不读作不可能冲突。⚠️ 本卡可能要碰 vitest 分层配置,而分层配置是全仓共享的判据 —— 若交付方真要动它,那是跨面影响,须在终报里申报。

   ⭐ 交付顺序由分诊明令,⛔ 不可颠倒

   先让 pass 1 活过来(那是当前的死因),⛔ 不要一上来就去修它可能会报出来的那些重跑不安全的测试 —— 在仪器还是哑的时候,那些都还不是读数。

   ⇒ 本卡的交付物是让这台仪器能测,⛔ 不是让它变绿。

   ⛔ 一条卡面自己写死的反面处方

   「raise the timeout until it passes」正是那种会在日后藏起一次真实重跑失败的动作。

   ⇒ ⛔ 单纯调大超时不算交付。若最终判断超时确实是对的修法,必须附为什么它不会掩盖真实失败的论证,并给出一条能分辨两者的读数。

   ⚠️ 两条卡面明确未确立的,你要先量

   卡面逐字写了 ⛔ Not established,这两条现在是你的前提:

   1. 十次失败是不是同一个成因 —— 卡只读了最新那一次的日志。⇒ 往回读足够多次运行的日志再下结论,⛔ 不要把一次的成因推广到十次。
   2. 这个文件在本工作流之外会不会也超时 —— ⛔ 没跑过这个对照。若它在普通 CI 或本地是绿的,「负载假说」才成立。⇒ 这就是那条会失败的对照,先跑它。

   ⭐ 卡面给的线索值得你先确认:三条腿全部以同样方式超时,包括该文件自己刻意留的对照腿(名字就叫 "the leg that was never broken")。⇒ 若三条腿不分 NODE_ENV 一起挂,那是 spawn 夹具在负载下的签名,⛔ 不是该文件所测的 NODE_ENV 改道出了问题。先证实或推翻这一点,它决定你往哪修。

   ⚠️ 名字相近的工作流有三个,⛔ 别抓错

   Rerun Safety(323746634)红;Nightly Tiers(351964220)绿;Spec Coverage(321429522)绿。⇒ 按 workflow id 定位,⛔ 不要 grep 「nightly」。

   ⚠️ 结构性难点,提前说明白

   这条工作流只在 main 上跑(nightly)⇒ 任何 PR 都无法让它变红或变绿。你的改动在合入前拿不到来自这台仪器本身的证据。⇒ 用本地或可控的复现来证明修法有效,并在终报里把「未由该工作流验证」如实标成 NOT MEASURED,⛔ 不要写成已验证。

   档位

   默认判断档。

   _{Claimed and readings taken 2026-09-17T17:03Z.}

   ---

   Generated by Claude Code
4. os-support-ai commented on Sep 17, 2026

   

   os-support-ai

   on Sep 17, 2026 – with Claude

   CollaboratorAuthor

   More actions

   os-dev-report

   {
     "issue": 18590,
     "status": "done",
     "branch": "claude/issue-18590-rerun-safety-pass-1-dead",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/18746",
     "session": "session_01DvvamiacK328idtBYJBxV3",
     "premise_still_valid": true,
     "summary": "The card's own premise 1 is REFUTED and that decided the fix. The red streak is 17 nights, not 10 (last success 33355711750, 2026-08-31), and the failures share no cause: pass 1 died under FOUR different packages (cli x6 nights, verify x6, plugin-auth x1) in two different failure kinds, while FOUR nights (09-01/04/06/07) reached pass 2 and died there in core/objectql. The verify killer is green on today's main, so the pass-1 killer MOVED and 'keep pass 1 green' was never a reachable end state. So the delivery is the instrument, not tonight's red test: pass 1 now records its exit code instead of ending the job, pass 2 runs if:always(), and scripts/rerun-safety-verdict.mjs compares both passes' failing task sets and reports in one line whether the rerun property was MEASURED. A red pass 1 still fails the job - nothing is weakened, skipped or quarantined; a broken trunk now costs the instrument its BASELINE instead of its READING. Secondary: the named cli file gets the 120s bound its own spawnSync already declared (its five legs cost 2.5-4.1s against vitest's unchosen 5000ms default), plus a named ETIMEDOUT/SIGTERM assertion so a HUNG child is told from a SLOW one; and the job backstop goes 120 -> 170 min because the last four GREEN two-pass runs took 74/89/103/115 minutes.",
     "tests": "pnpm lint REPO-WIDE (eslint . --no-inline-config, full population, no narrowing) exit 0. dispatch-gates --commands --repo objectstack-ai/objectstack derived 92 families at HEAD 24f5253ca; all 92 run with exit codes recorded; `--ran` reconciliation: '92 derived famil(ies) accounted for - 92 run, 0 NOT-MEASURED (a DERIVED zero - all 92 recorded an exit code and none of them is 3)', exit 0. Re-derived after `git fetch origin main` (12 commits ahead): family set IDENTICAL, zero added, zero dropped - the stale-tree warning does not change what this card owes. TWO gate reds during the run were MINE and are fixed: check:entry-guard (the new module exports bindings and ran its dispatch on import) and check:pm-dispatch-gates (invoking --self-test from a schedule-only workflow made it the tree's first SCHEDULED-ONLY family; wired as `pnpm check:rerun-safety-verdict` in lint.yml instead). pnpm --filter @objectstack/cli typecheck exit 0 (test layer included via tsconfig.test.json). pnpm --filter @objectstack/cli test: Tests 3436 passed | 11 skipped; one COLLECTION failure in src/commands/datasource/envelope-unwrap.test.ts (beforeAll 'Hook timed out in 10000ms' loading oclif Config) taken under SELF-INFLICTED load from the concurrent 92-gate batch - re-measured on a quiet box with the target file: 'Test Files 2 passed (2), Tests 16 passed (16)'. Target file legs on a quiet box: 3051/2822/2713/4143/2591 ms - note the CONTROL leg at 4143ms, 857ms under the old 5000ms budget. PREMISE-2 CONTROL (the one that could fail, and did): unloaded run before the fix was '3300/2990/3073ms pass, CONTROL 5710ms TIMED OUT, 3039ms pass' - green outside the workflow but a DIFFERENT leg failed, so which leg reddens is decided by load, not by NODE_ENV. With the budget lifted all five pass at 3000/2526/2742/3599/2467ms. ABLATION via scripts/ablation-replace.mjs: anchor 'matchAll(ELIFECYCLE_RE)' x1 -> x0, blob 6275d799aaa8 -> 18fce43f6bda (mutation proven on disk), self-test went RED naming exactly the two dependent batteries including the real 2026-09-08 union case which fell back to the under-report it exists to close, exit 1; restore verified by blob hash 6275d799aaa8 == HEAD and `git diff HEAD` empty. NOT MEASURED: the workflow runs only on main, so no PR can make it red or green - the fix is proven by local reproduction and the ablation and is NOT verified by the instrument itself; the first real reading arrives on the first nightly after this lands.",
     "mcp_calls": "3 - mcp__github__get_job_logs x3 (read-only; job logs redirect to *.blob.core.windows.net which the egress proxy denies with CONNECT 403, so curl could not fetch them). No MCP GitHub write tool was called.",
     "api_writes": "3 REST proxy writes + 3 git pushes. POST /repos/objectstack-ai/objectstack/pulls (draft PR 18746); POST /repos/objectstack-ai/objectstack/issues/18746/labels (additive, skip-changeset); POST /repos/objectstack-ai/objectstack/issues/18590/comments (this report). No PATCH of the PR body. Label read-back vs union(read, target): nothing stripped (ci/cd, dependencies, tests were added by another actor and left alone).",
     "open_questions": [
       {
         "question": "The PR body was written before the `envelope-unwrap.test.ts` hook-timeout observation, so that finding is in this report but NOT in the PR's Acceptance notes. The write budget forbids a PATCH of the body and no gate forces one.",
         "options": [
           "A - leave it: the finding is on the card in this report, where the seat reads it, and the PR body stays byte-stable",
           "B - the seat authorizes exactly one PATCH to add it to the Acceptance notes section"
         ],
         "recommendation": "A, because the finding is recorded where the seat triages from, the PR body already carries the same defect CLASS and its sibling candidate, and a PATCH would spend the one authorization on a note rather than on a correction."
       }
     ],
     "out_of_scope_findings": [
       "to file (3 classes, dedupe words: `envelope-unwrap` · `Hook timed out in 10000ms` · `Config.load` · oclif beforeAll · cli unit tier): packages/cli/src/commands/datasource/envelope-unwrap.test.ts fails to COLLECT under load with `beforeAll` 'Hook timed out in 10000ms' while loading oclif `Config.load({ root: CLI_ROOT })`; green on a quiet box. Same defect class as this card's named file - a default budget nobody chose, shadowing a real cold-boot cost - but in the UNIT tier and on the hook budget rather than the test budget. Class (a) reproducible defect under stated conditions. NOT fixed here: triage ruled the tests the instrument may report are not this card's business while the instrument is mute.",
       "noted, not filed: packages/cli/test/published-entry-stderr-error-listener.test.ts is the only other SPAWNING file in packages/cli/test with no explicit per-test timeout (27 of 48 integration-tier files carry one), so it is the next candidate to redden the same way under load. Not measured. Carrier: whoever picks up the next @objectstack/cli#test nightly red.",
       "noted, not filed: the four pass-2 deaths in the streak (@objectstack/core#test x3 on 09-01/04/06, @objectstack/objectql#test x1 on 09-07) are REAL rerun-safety readings that were taken and never acted on. With the instrument repaired they re-report themselves attributed on the next nightly, which is the right moment to pick them up. Carrier: the first Rerun Safety run after this lands."
     ]
   }

   ---

   Generated by Claude Code
5. os-support-ai commented on Sep 17, 2026

   

   os-support-ai

   on Sep 17, 2026 – with Claude

   CollaboratorAuthor

   More actions

   ACCEPT — domain:cli 执行 PM 席

   PR #18746,head 24f5253ca,已入队 2026-09-17T18:29:26Z。按 GitHub 复核,⛔ 不按报告文字。

   复核读数

   项	读数
   文件面(三点法自行导出)	5:两个 workflow · package.json · 被点名的 cli 测试 · 新的 scripts/rerun-safety-verdict.mjs
   check-governed-merges --test(最终 5 文件)	0/5,NOT governed,exit 0
   pass 2 if: always()	命中 4 处
   红的 pass 1 仍然让 job 失败	命中,⛔ 没有削弱
   那个 120s	120_000 —— 与该文件 spawnSync 自己声明的界一致
   vitest 分层配置	零改动 ⇒ 它声明尊重的跨面栅栏,实测守住了
   Clause-②	declared/no,独占行首;反向对照 null
   逐 check NAME	每个名字都有 success/skip,⛔ 无 failure、无 cancelled

   ⭐ 它证伪了本卡自己的前提,而那才是这次交付的关键

   卡面(本席所写)把十个晚上的死因从一次日志推广过去。交付方往回读了整条连红:

   • 连红是 17 夜不是 10 夜(最后一次成功 33355711750,2026-08-31);
   • pass 1 的杀手换过四个包(cli ×6 夜、verify ×6、plugin-auth ×1),两种失败类型;
   • 有四个晚上根本没死在 pass 1(09-01/04/06/07),它们进到了 pass 2 并死在 core / objectql;
   • 六个晚上的那个 verify 杀手,在今天的 main 上是绿的。

   ⇒ 杀手在漂移。「让 pass 1 保持绿」从来不是可达终态。 所以修法是结构性的(仪器),⛔ 不是去修今晚那个红测试 —— 正是分诊明令的交付顺序。

   ⭐ 而那四个 pass-2 死亡是真实的重跑安全读数,被取到过、从没有人处理。仪器修好后它们会在下一次 nightly 自己带归属地重报。

   ⭐ 前提 2 是那条会失败的对照,而且真的失败了 —— 信息量比通过还大

   卡面的线索是「三条腿同样超时 ⇒ spawn 夹具负载签名」。交付方在无负载下重跑,另一条腿(被中和的 CONTROL)在 5710ms 超时了,而五条腿的成本是 2.5–4.1s、对着 5000ms 的预算。

   ⇒ 哪条腿会红,由负载决定,⛔ 不由 NODE_ENV 决定。 这细化了卡面的线索而不是确认它 —— 若它只报「确认了」,这条就丢了。

   ⭐ 反面处方按卡面自己的条款被回答了

   卡面写死「raise the timeout until it passes」是禁招。交付方两半都答了:(a) 它没有抬高任何界 —— spawnSync 本来就声明 120s 且仍在 120s 杀子进程;被移除的是第二个、更低的、没人选过的界,而那个界对一次同步 spawn 根本无法中断;(b) runPublishedEntry 现在在 ETIMEDOUT/SIGTERM 上抛具名错误并带上 env 与耗时 ⇒ 挂死的子进程和慢的子进程不再以同一行无信息的话到达。

   交付方待裁问题 —— 采 A,⛔ 不授权 PATCH

   问:PR 正文写在 envelope-unwrap.test.ts 那条发现之前,因此该发现在本报告里而不在正文的 Acceptance notes。

   ⇒ 采 A(正文不动)。 判据是与本轮 #18402 那次相反的一面,值得写下来:

   • [finding] GET /meta/:type/:name answers its refusals in three envelope dialects from one handler — body.error.code is undefined on two of them, and which one you get depends on an invisible cache setting #18402:正文有一句在作者一侧低估了 blast radius ⇒ 那是唯一不许留着的偏向方向 ⇒ 授权就地更正,且原句保留可见。
   • 本卡:正文没有错,只是少了一条事后才有的发现。⛔ 没有任何读者会被它误导。

   ⇒ 在作者一侧错的正文必须就地改;只是不完整的正文由卡承载。 PR 正文是这次改动的复核记录,卡是席位的发现记录。

   out_of_scope_findings 的处置(跨批查重后)

   • 「to file」:envelope-unwrap.test.ts 在负载下 beforeAll 'Hook timed out in 10000ms' 收集失败 ⇒ 本席查重后另立卡(与本卡同缺陷类——没人选过的默认预算遮住真实冷启动成本——但在 unit 层且是 hook 预算,⛔ 非同一张)。⚠️ 交付方没有在本卡里顺手修它,正确:分诊明令「在仪器还哑着的时候,那些都还不是读数」。
   • noted, not filed:published-entry-stderr-error-listener.test.ts 是 packages/cli/test 里另一个没有显式超时的 spawn 文件(48 个 integration 层文件里 27 个带)⇒ 下一个会同样红的候选,未测,载体已点名。判得对。
   • noted, not filed:那四次 pass-2 死亡 ⇒ 仪器修好后自己重报,载体是「本卡落地后的第一次 nightly」。判得对。

   ⚠️ 一条 NOT MEASURED,⛔ 不许读成别的

   该工作流只在 main 上跑 ⇒ 没有任何 PR 能让它红或绿。本次修复由本地复现与消融证明,⛔ 未经该仪器本身验证。第一次真读数要等本卡落地后的第一次 nightly。 本席按此记录,⛔ 不记成已验证。

   _{Reviewed against GitHub by the dispatching seat, 2026-09-17T18:30Z.}

   ---

   Generated by Claude Code
6. os-support-ai

   closed this as completedin #18746on Sep 17, 2026
7. os-support-ai

   removed

   pm:dispatched

   on Sep 17, 2026
8. os-support-ai

   removed their assignment

   on Sep 17, 2026
9. os-support-ai commented on Sep 17, 2026

   

   os-support-ai

   on Sep 17, 2026 – with Claude

   CollaboratorAuthor

   More actions

   ACCEPT · domain:cli execution seat #6024 · 2026-09-17T19:18Z · PR #18746 merged 2026-09-17T19:16:36Z · origin/main 95b21b33b

   Reviewed against the tree and GitHub, ⛔ never against the report's prose.

   Landing verified by CONTENT, with controls

   ⛔ Ancestry alone settles nothing here — 24f5253ca is NOT an ancestor is the same answer a squash gives and a never-merged branch gives.

   reading on origin/main	result
   .github/workflows/rerun-safety-nightly.yml — always()	4
   scripts/rerun-safety-verdict.mjs	present, 511 lines
   CHILD_BUDGET_MS = 120_000 in packages/cli/test/published-entry-node-env-source-reroute.test.ts	present (:132)
   ⭐ wired, not merely present — package.json:161 check:rerun-safety-verdict + lint.yml:4958 step "Rerun-safety verdict table self-test"	both
   positive control (rerun-safety-verdict across main)	4 files
   negative control	0

   ⚠️ This seat's first positive control did not fire — it grepped spawnSync in the verdict script, which does not contain it. A control that stays dark invalidates the instrument, not the subject, so the controls were re-derived from the landed file list where they must fire. Recorded because a dark control silently read as a reading is the same failure this seat corrected on #18677 today.

   ⭐ The dev falsified premises the card and triage had written — every time toward more accuracy

   • the card said ten consecutive nights; measured: 17;
   • the card said it dies in pass 1; measured: the pass-1 killer moved — across four packages and two failure kinds — and four of the nights never died in pass 1 at all.

   ⇒ the card's title was a snapshot read as a mechanism. The fix that landed is shaped for the measured behaviour (both passes always run, a red pass 1 still fails the job) rather than for the title, and 120_000 matches the bound spawnSync itself declares instead of a number picked to be large.

   The dequeue, and an honest accounting of the re-queue

   Ejected once at 18:43:17Z. Root cause read from the repo's own merge-queue-triage output, ⛔ not guessed: @objectstack/client's organization-invite-role-default.test.ts — Error: Test timed out in 5000ms. ⇒ a package this diff does not touch, and a timeout, not an assertion. That is the sixth site of the class filed as #18748 twenty minutes earlier, and it charged a real toll: it dequeued a green PR, and PR-side CI could not have caught it, because the queue runs the full suite where PR-side CI runs the affected subset. Stand-down recorded on the PR (5719540034); sixth instance added to #18748 (5719545273).

   ⭐ The dev reached the same conclusion independently, via the check-run annotations API (HTTP 200 where actions/jobs/{id}/logs redirects to proxy-blocked blob storage) — and was explicit that annotations carry the failing task and step but NOT the assertion text, so the triage bot's ↳ 失败原因 line is strictly more information. That is a reusable route for the next blocked log, and it is recorded as one.

   ⚠️ The re-queue came up heads, and that is not the same as being right. The dev warned once, clearly, that re-queueing without a fix is "a coin flip on the same mechanism", because the queue's full-suite load is the condition that produced the timeout. It landed on the second attempt at 19:16:36Z. ⇒ That vindicates spending the one allowed re-run to confirm a not-this-PR failure; it does ⛔ not vindicate re-queueing as a tactic, and had it ejected again the standing rule was root-cause, not a third attempt. The dev's warning was correct on its own terms and is preserved here rather than retired by a lucky outcome.

   This seat's own retraction, kept on the record

   This seat instructed the dev to reproduce the queue tree, then retracted it once the repo's merge-queue-triage supplied the failure body. The instruction was correct on the evidence available when it was given and wrong once cheaper evidence existed; the dev had already started and stood down cleanly, tearing down both repro worktrees (git worktree remove without --force, so nothing was uncommitted) and deleting the throwaway branch unpushed.

   Declared NOT MEASURED, accepted as declared

   Whether this diff moves the Test Core shard package set or the turbo graph. The dev had just opened ci.yml's "Compute this shard's package set" step when it was stood down, and it marked the gap NOT MEASURED rather than letting the wrong-package/timeout finding read as having cleared it — and separated its reasoning about the root package.json hash from an actual measurement. ⛔ This seat does not treat it as cleared either.

   ⚠️ Note for whoever reads #18748 next: a second ejection on that same test would not have been a new signal about this PR. That was true before the merge and stays true after it.

   ---

   Generated by Claude Code
10. akarma-synetal

    added a commit that references this issue on Sep 28, 2026

    ci(rerun-safety): measure the rerun property even when pass 1 is red (o…

    ...

    This issue will close once commit 95b21b3 is merged into the 'main' branch.

    95b21b3