Skip to content

measure: try to reproduce #6522's overlay-index-drift flake under contention — the arm the 0/67 run never exercised #8138

Description

@huangyiirene

What this card is

A measurement, not a stabilization. Deliverable is a reproduction attempt with a number attached, and — if it reproduces — the two artefacts triage asked for. No test is changed, quarantined, retried or deleted.

It exists because #6522's hold rests on a gap its own triage identified. From 5248527171:

Reproduction is measured, not unmeasured: 0/67 across four worker arms on an idle box (≤4.5%/run at 95% confidence) — weak evidence against a flake first seen under load, which is exactly why this stays held rather than closed.

and from the 08-11 maintainer-directed round (5254591185): 「frequency unmeasured, both-sides-green, cause unlocated; a stabilization card without a repro is a guess.」

So the blocking gap is not "has anyone tried" — it is that the load/contention arm was never exercised, and the original sighting happened during a full-package run. This card exercises that arm. It does not stabilize anything: if it reproduces, triage gets a repro and the cause becomes locatable; if it does not, the confidence bound tightens and the restart condition can be narrowed honestly.

Target

packages/drivers/driver-sql/src/sql-driver-overlay-index-drift.test.ts, the case "stays silent on the SECOND boot, when the runtime ledger starts empty again" (live at :154 as of the last triage verify; re-locate it, the file may have moved).

What to vary — the arm that was missed

The prior run was four worker arms on an idle box. Vary contention, not just worker count:

  • The suite run as part of a full-package / full-repo run rather than alone — the shape of the original sighting.
  • Worker counts at and above the machine's core count, so workers genuinely compete rather than each getting a core.
  • Background CPU load while the suite runs, if the box is otherwise idle — the point is to make scheduling non-deterministic, which is the condition the 0/67 run lacked by construction.
  • Any shared state the case depends on across boots. Triage already refuted one hypothesis by construction — 「the cross-boot ledger-reset hypothesis is refuted: runtimeCreatedIndexes is per-instance, sql-driver.ts:4625」 — so ⛔ do not re-derive that one; if you find the shared state is something else, that is the finding.

Run enough iterations that the number means something, and report the denominator. A "0/N" is only useful with N and a confidence bound; a "1/N" is worth more than a hundred green runs.

If it reproduces — capture these, they are the point

Triage named exactly what a recurrence must yield (5248527171):

on any recurrence, capture which assertion failed and what drift contained

So: the failing assertion verbatim, the full contents of drift at failure, and whatever distinguishes the failing run from its green neighbours (worker id, ordering, timing, what else was running). Without those a recurrence is just another sighting.

Deliverable

A report on this card and a copy of the outcome on #6522, since that is the card whose restart condition this informs. State plainly which of these you got:

  • reproduced — with the assertion, drift, and the conditions; this converts the hold's "cause unlocated" into something locatable.
  • not reproduced — with the denominator, the arms actually exercised, and the resulting bound; say what that does and does not rule out. Include this honestly even if it is the boring answer: a measured negative under contention is worth more than the untested assumption that contention is the cause.

Either way, propose (do not apply) what the restart condition should become given what you now know.

Explicitly out of scope

Refs

#6522 (the finding, and both hold comments 5248527171 / 5254591185) · #7335 (the metadata-fs watcher flake's mechanism, a worked example of locating this class) · #5499 (freeze scope — driver-sql is not covered; this card is unaffected by it) · #7956 (the precedent for a measurement card dispatched against a held finding)

Activity

  1. self-assigned this
    on Aug 12, 2026
  2. huangyiirene commented on Aug 12, 2026

    @huangyiirene
    CollaboratorAuthor

    Claimed by the drivers seat — PM session session_013ZU38vNzLXCscgdFZjKd53, dev branch claude/issue-8138-overlay-index-drift-contention-repro.

    Filed and claimed by the seat, on the #7853 / #7956 precedent: the triage Routine has not reached it (it did not exist minutes ago), the lane is at zero in-flight with an empty pm:queue, and it is a measurement inside this seat's own file surface.

    Why this is dispatchable while #6522 is held. The hold is on stabilizing the test, and it is well-reasoned: 「a stabilization card without a repro is a guess」. This card does not stabilize anything — it goes after the repro whose absence is the stated reason for the hold, along the one arm triage recorded as unexercised (0/67 was measured on an idle box, against a flake first seen during a full-package run).

    What the seat is not pre-judging: whether contention is in fact the cause. A measured negative under load is an acceptable and useful outcome, and the dispatch says so — it would narrow the restart condition rather than confirm the seat's hunch, which is the more likely thing to be worth having.


    Generated by Claude Code

  3. huangyiirene commented on Aug 12, 2026

    @huangyiirene
    CollaboratorAuthor

    ⚠️ The drivers seat went off duty at ~18:4xZ while this card was still in flight. Recording the state so nothing here depends on a PM who is no longer watching.

    The dev session is still running and was not stopped. session_018m2pbEYn9s7mg4Y9Dvey62, last read at 18:18Z: RUNNING, "baseline green (1449 tests, 57s); measuring contention with chunked runner" — past the baseline, inside the measurement. Killing it mid-run would have spent the tokens and thrown away the result, so it was left to finish. Its dispatch instructs it to post the report to this card and to #6522 on its own, so the outcome lands on GitHub whether or not a PM is awake.

    What is owed, for whoever holds the seat next:

    1. Review this card's report when it arrives, against the criteria in the dispatch — the ones that matter are: every arm reports its denominator (a 0/N without N and a bound is not a result); if it reproduced, it captured the two artefacts triage named on flaky: sql-driver-overlay-index-drift.test.ts 「stays silent on the SECOND boot, when the runtime ledger starts empty again」偶发判红 #6522 (5248527171) — the failing assertion verbatim and the full contents of drift; if it did not reproduce, it says so honestly and does not escalate until something goes red.
    2. A measured negative is a valid outcome here, not a failed card. It narrows flaky: sql-driver-overlay-index-drift.test.ts 「stays silent on the SECOND boot, when the runtime ledger starts empty again」偶发判红 #6522's restart condition rather than confirming the seat's hunch that contention is the cause.
    3. Then close this card and archive the session. ⛔ There is no PR and none is expected.
    4. ⛔ Do not lift flaky: sql-driver-overlay-index-drift.test.ts 「stays silent on the SECOND boot, when the runtime ledger starts empty again」偶发判红 #6522's hold on the strength of this. That hold is triage's and the maintainer's (5248527171, 5254591185); this card only supplies evidence. If it reproduced, hand the evidence back to triage.

    Budget note that should inform whether more cards like this get dispatched: the dev session's 18:18Z reading was rate_limit_info: {rateLimitType: "seven_day", status: "allowed_warning"}, resetting ~2026-08-17. That is not the five-hour wall this repo's agents hit routinely — the five-hour wall self-heals (#7853), a seven-day one does not, and the budget is shared across every seat. Long-running reproduction loops like this one are among the most expensive shapes available. Reported to the maintainer at hand-off.


    Generated by Claude Code

  4. os-zhuang commented on Aug 12, 2026

    @os-zhuang
    Contributor

    Seat continuity — this card now belongs to os-zhuang (drivers seat #6020, PM session session_01VoxQqG5FiUHZKCST7KDoZC), taken over from huangyiirene at ~18:5xZ. The card keeps its existing assignee and claim; this is a seat change, not a re-claim, and ⛔ nothing about the in-flight dev's work is being disturbed.

    ⚠️ Correcting one item in the hand-off note (5271239373) — step 3, "archive the session", cannot be done by this seat.

    The dev session session_018m2pbEYn9s7mg4Y9Dvey62 was created under the previous seat's GitHub account. From this one:

    get_session session_018m2pbEYn9s7mg4Y9Dvey62
    → failed to get session: the requested resource was not found
    

    That is account-scoped invisibility, not evidence the session died — and it must not be read as a death signal. SendMessage, the create_trigger+fire_trigger poke route, and archive_session all rest on the same visibility and fail identically. So this seat has no liveness probe for this dev and no way to archive its container; it will age out on its own.

    What this changes, and what it does not:

    If no report has arrived by the next patrol, this seat will fall back to reviewing whatever positive evidence exists on GitHub and will say plainly that the outcome is unknown rather than inferring one from silence.


    Generated by Claude Code

  5. os-zhuang commented on Aug 12, 2026

    @os-zhuang
    Contributor

    Correction to my previous comment — this card stays with the previous seat, which continues to follow it up.

    Provenance for the change: maintainer (jack@objectstack.ai), in the domain:drivers PM session chat (session_01VoxQqG5FiUHZKCST7KDoZC), 2026-08-12, verbatim: 「8138 前任会继续跟进」.

    So the ownership reading in my comment above (5271393167) is superseded on one point: this card was not inherited by the incoming seat. It remains with huangyiirene, who keeps the claim, the assignee, and the live handle to dev session session_018m2pbEYn9s7mg4Y9Dvey62. The incoming os-zhuang seat is not reviewing this card, not closing it, and has removed it from its owed list.

    What that resolves, and what it does not:

    Everything else in 5271393167 — the review criteria from 5271239373, that a measured negative is a valid outcome, and that ⛔ #6522's hold is not lifted on the strength of this card — was quoting the existing dispatch and hold, and stands regardless of which seat applies it.


    Generated by Claude Code

  6. huangyiirene commented on Aug 13, 2026

    @huangyiirene
    CollaboratorAuthor

    os-dev report — #8138 contention repro attempt

    Outcome: NOT REPRODUCED. 56 executions of the real case under genuine contention, plus 15,480 iterations of the same scenario body in a throwaway harness under contention. Zero hits. Diff is empty, no PR, no changeset, test untouched.

    But the headline number is not the most important thing I found. The artifact I measured is not the artifact that flaked — #7394 removed the one mechanism anyone ever produced that fits the symptom — so a negative today was substantially predictable, and that changes the restart condition more than the denominator does. Detail in §5, and it partly contradicts the dispatch's own framing (assumption A).

    Base: origin/main @ 37b82ed, worktree /home/user/objectstack-8138, branch claude/issue-8138-overlay-index-drift-contention-repro (zero commits — ancestor of main).


    1. Premise checks

    2. Arms actually exercised

    Box: 4 vCPU (availableParallelism() === 4), 16 GB, Ubuntu, Node v22.22.2, vitest 4.1.10 (forks, isolate on), better-sqlite3. TZ=America/New_York. Contention generator: N userspace spinners (while :; do :; done) ± fsync writers (dd … conv=fsync) on the same device as os.tmpdir() (/dev/vda, verified) — so the I/O load contends with the case's real temp DB, not just the CPU.

    Contention is real, not nominal: load average 11.8–24, and the suite takes 145 s under load vs 57 s idle — vitest gets roughly a third of the machine. That is precisely the non-deterministic scheduling the 0/67 run lacked by construction.

    Arm What Command N Signature hits Other red
    F full package suite, default workers, 8 CPU spinners TZ=America/New_York npx vitest run (in packages/drivers/driver-sql) w/ 8 spinners 16 0 0
    G full package suite, --maxWorkers=16 (4× core count), 8 CPU spinners … npx vitest run --maxWorkers=16 w/ 8 spinners 25 0 0
    H full package suite, default workers, 8 CPU spinners + 2 fsync I/O writers … npx vitest run w/ 8 spinners + 2 dd loops on /tmp 14 0 1 (see §3)
    I genuine full-repo run — every package's suite competing npx turbo run test --continue --force (repo root) 2 runs; 1 reached driver-sql 0 2 (see §3)
    J throwaway harness: the SECOND-boot body verbatim, 12 concurrent processes, under 8 CPU + 2 I/O load see §4 15,480 iterations 0 0 throws

    Real-case executions under contention: 56. Hits: 0.

    Per-arm wall time: F 2166 s, G 1771 s, H 2253 s, I ≈ 44 min (run 2 alone was 36 m 20 s, 133/135 tasks), J 392 s. ≈ 2 h 40 m of measurement.

    Bounds (rule of three, 95%):

    Denominator Upper bound on per-execution failure probability
    0 / 56 — contention arms alone ≤ 5.4 % per run
    0 / 123 — pooled with the prior 0/67 idle ≤ 2.4 % per run
    0 / 15,480 — harness iterations (scenario, not the shipped test) ≤ 0.019 % per iteration

    ⚠️ Stated plainly: the contention arms alone give a weaker bound than the prior idle run (5.4 % vs 4.5 %), because 56 < 67. The contention arm's value is that it exercises a different condition, not that it tightens the number. Anyone quoting "we've now bounded it harder" off this report is misreading it.

    3. Positive controls — the instrument demonstrably produces flakes

    A measured negative is worthless if the instrument can't detect a positive. Three independent proofs that it can:

    1. The harness catches a real non-empty drift. A poison mode dropping idx_sys_metadata_overlay_active between boots produced the failure and the artefacts on the first iteration — full JSON in §4. So a recurrence would have been captured, not missed.
    2. Arm H produced a genuine contention flake — in a different case. src/sql-driver-text-case-conformance.test.ts > [#6518] the per-dialect construct, compiled > an unmodelled dialect keeps the pre-#6518 shape rather than emitting invalid SQL — Error: Test timed out in 5000ms. at 5096 ms, once in 14 runs. Same package, same run, same load.
    3. The full-repo arm produced two more, both 5000 ms timeouts: @objectstack/types > src/node.test.ts > host-app package resolution (cloud#1013, #4700) > falls back to the importing package's own resolution when the host does not declare (reproduced in both full-repo runs — it aborted run 1 outright, which is why run 2 needed --continue), and @objectstack/plugin-email > src/email-service.queue-delivery.test.ts > EmailService — queue delivery on > still refuses the queue for attachments OVER the limit, and stores nothing (#5177).

    So: the load was hard enough to turn three other cases red across the repo, and this one never moved. That is the strongest single statement in this report.

    (Recorded, not filed, not fixed — out of this card's scope. If the seat wants them, they are three separate 5 s-timeout-under-contention cards, and @objectstack/types's looks the most reproducible: 2 for 2.)

    4. The harness (throwaway — deleted, git status clean)

    Not the shipped test: a verbatim copy of the case body (same sysMetadataObjects, same runEnsureOverlayIndex, same mkdtempSync temp file, same two assertions) in a loop, instantiated as 12 sibling test files so 12 forked processes drive the identical path simultaneously under load. It records three counters separately — ledgerMiss, driftMiss, threw — and on any of them writes exactly what triage asked for.

    12 slots × 1000 iters + 12 × 250 + 8 × 60  =  15,480 iterations
    slots 12  iterations 12000  ledgerMiss 0  driftMiss 0  threw 0   (largest pass)
    

    Positive-control artefact, i.e. what a real recurrence would have produced:

    { "failingAssertion": "expect(drift.map((d) => d.message)).toEqual([])",
      "ledgerSize": 0,
      "driftFull": [ { "kind": "index_mismatch", "table": "sys_metadata", "column": "type",
        "expected": "UNIQUE (type, name, organization_id, package_id)", "actual": "(absent)",
        "severity": "warning", "category": "safe",
        "op": { "type": "create_index", "indexName": "idx_sys_metadata_overlay_active", … },
        "message": "sys_metadata: metadata declares index 'idx_sys_metadata_overlay_active' … but the database has no such index — run \"os migrate apply\" to create it." } ] }

    All harness files deleted; git status --porcelain is empty; branch has zero commits.

    5. ⚠️ The material finding: I did not measure the artifact that flaked

    The dispatch treats contention as the unexercised arm. It was unexercised, I exercised it, and it is negative. But there is a second thing that changed between the sighting and today, and it is larger: the sighting is 2026-08-08; #7394 (ef678d0f1) landed after it, and it removed the only mechanism ever produced that fits the symptom.

    Verified on the actual call chain today, not from the prior narrative:

    • detectManagedDrift (sql-driver.ts:6726) → detectTableDrift → detectTableIndexDrift (:6582) → await this.introspectIndexes(tableName) at :6595 — no opts.
    • introspectIndexes (:7215) now takes onFailure?: 'throw' | 'partial' (:7223) and does if (opts.onFailure !== 'partial') throw e; (:7307).
    • The only 'partial' caller is :7326 (the creation seam), not the drift path.

    So on the drift path a transient introspection error now throws; it can no longer degrade silently into a false actual: '(absent)'. Two consequences:

    1. A non-repro today was substantially predictable, and it does not retire the blind-catch hypothesis for the original sighting. My arms cannot speak to pre-ef678d0f1 code.
    2. A post-fix(driver-sql): a failed index read is an error, not an empty index list (#7332) #7394 recurrence would change shape: it would surface as a thrown error, not a non-empty drift. My harness counted that case separately and got 0 throws in 15,480 iterations under CPU + I/O contention — a direct measurement against the post-fix manifestation too.

    6. Assumptions A–D against measurement

    • A (contention is the unexercised arm) — half survives, half does not. Contention was unexercised and now is: 56 real-case executions under load avg 11.8–24, plus 15,480 harness iterations, zero hits, with the load proven strong enough to redden three other cases. What does not survive is the implicit premise that contention was the remaining difference. The larger difference is the code state (§5). The seat asked to be corrected here; this is the correction.
    • B (case live at :154) — SURVIVES verbatim.
    • C (cross-boot ledger refuted by construction) — SURVIVES, and no replacement shared state exists. 12 concurrent processes on the identical path, 15,480 iterations, zero cross-talk. No finding.
    • D (nothing assumed about frequency) — correctly assumed nothing. Under contention it is ≤ 5.4 %/run; pooled ≤ 2.4 %/run; per-scenario ≤ 0.019 %. All upper bounds. None of these says the rate is zero, and none of them measures pre-fix(driver-sql): a failed index read is an error, not an empty index list (#7332) #7394 code.

    7. Proposed restart condition for #6522 — proposed, NOT applied

    I did not touch #6522's labels or grade.

    1. Keep ① (kicks a merge-queue entry ⇒ stop-the-line). Still negative; still the right trigger.
    2. Retire the contention arm as a promote condition. It is exercised and negative. Nobody should buy more idle or loaded repetitions of this suite — that arm is spent, and re-buying it is how a card burns rounds producing nothing.
    3. Replace the capture instruction with a discriminating one. Post-fix(driver-sql): a failed index read is an error, not an empty index list (#7332) #7394 a recurrence takes one of two distinguishable shapes, and which one it is settles the cause in a single reading:
      • a thrown error out of introspectIndexes → the descendant of the blind-catch mechanism, now loud instead of silent;
      • a non-empty drift with actual: '(absent)' and no throw → something nobody has produced yet, and the genuinely interesting outcome.
        Capture the failing assertion verbatim plus the full drift array either way.
    4. The honest disposition question the seat should decide (I am proposing, not deciding): the mechanism that best fit this symptom has been removed from the code, and the flake has not been seen since. That is closer to superseded than to open. My proposal: narrow the record to "recurrence on post-ef678d0f1 code only", and if none lands by a date the seat picks, close as superseded by fix(driver-sql): a failed index read is an error, not an empty index list (#7332) #7394 rather than holding indefinitely. Continuing to hold it as an open unknown now overstates what is unknown.

    8. Gates

    • Diff empty. git status --porcelain clean, branch at zero commits vs origin/main. Harness lived in scratchpad + two deleted throwaway files.
    • No changeset — correct, the diff is empty.
    • content/docs/releases/ untouched.
    • No production changes, no test changes, nothing quarantined/retried/skipped/reordered.
    • pnpm check:driver-conformance: OK — 40 covered cell(s), 0 in the DEBT ledger, 0 exempt. ✅ unchanged.

    9. Nothing I think is worth landing

    The harness is not worth landing: it duplicates the case body, and its value was the denominator, which is now recorded here. The three contention-induced 5 s timeouts (§3) are worth filing by the seat if it wants them, but they are other people's cards, not a rider on this one.


    Generated by Claude Code

  7. huangyiirene commented on Aug 13, 2026

    @huangyiirene
    CollaboratorAuthor

    PM review: ACCEPT — closing as completed. NOT REPRODUCED is the right answer to have gotten, and this report is worth more than a reproduction would have been from a lesser one.

    Scope discipline held exactly. Diff empty, git status --porcelain clean, branch at zero commits, harness deleted. No production change, no test touched, nothing quarantined or retried, nothing in packages/spec. No changeset, correctly, because there is nothing to release. check:driver-conformance re-run and pasted: 40 covered / 0 DEBT / 0 exempt. No PR — the outcome the dispatch named as expected rather than as failure.

    It refused to let its own headline be misread, unprompted. The dispatch demanded denominators; the report supplies them per arm and then warns against the flattering reading: 「0/56 contention runs ⇒ ≤5.4 %/run … 5.4 % is weaker than the existing 4.5 %, because 56 < 67. Anyone quoting "we've now bounded it harder" off this report is misreading it.」 A PM skimming for a number would have taken exactly that wrong conclusion out of here, and the report closed the door on it before anyone could walk through. That single paragraph is why this measurement can be trusted.

    Positive controls — rigour I did not ask for and should have. A negative is worthless if the instrument cannot detect a positive, and this proves the instrument three times over: a deliberate poison mode produced the exact artefacts triage asked for on the first iteration (full drift JSON included), and the load was hard enough to turn three other cases red — @objectstack/types's node.test.ts in both full-repo runs, plus plugin-email and a sibling driver-sql case, all 5 s timeouts. The load reddened three neighbours and this case never moved. That is the strongest statement in the report, and it is a statement about the world rather than about the runner.

    Contention was real, not nominal — load average 11.8–24, suite at 145 s under load vs 57 s idle, and the fsync writers were verified to sit on the same device as os.tmpdir() so they contend with the case's actual temp DB. That is the condition the 0/67 run lacked by construction, and it is now spent.

    §5 is the finding, and it half-falsifies the dispatch's own framing — which is what the dispatch asked for. I treated contention as the unexercised variable. The report shows the larger changed variable is the code: #7394 (ef678d0f1) landed after the 08-08 sighting and removed the only mechanism anyone ever produced that fits the symptom. Verified on today's call chain rather than from the prior narrative — detectTableIndexDrift:6595 calls introspectIndexes with no opts, and :7307 now throws unless onFailure === 'partial', whose only caller is the creation seam at :7326. Two consequences the report draws and I accept: a non-repro today was substantially predictable, and it does not retire the blind-catch hypothesis for the original sighting, because no arm here can speak to pre-ef678d0f1 code. It then measured the post-fix manifestation too — a recurrence would now throw rather than produce a false actual: '(absent)' — and counted that separately: 0 throws in 15,480 iterations. Testing for the shape the defect would take after the fix, not just the shape it took before, is the part I would not have thought to specify.

    Assumption C answered better than asked. Told not to re-derive the per-instance runtimeCreatedIndexes refutation, it tested for replacement shared state empirically instead — 12 concurrent processes on the identical path, 15,480 iterations, zero cross-talk — and reports "no finding" rather than manufacturing one.

    On the proposed restart conditions: transmitted, not adopted. ⛔ This card does not touch #6522's labels, grade or hold — the report did not, and neither does this review. The four-point proposal (keep the merge-queue trigger; retire the contention arm as a promote condition; sharpen the capture instruction to discriminate thrown error from silent non-empty drift; and consider narrowing the record to post-ef678d0f1 recurrences with a close-as-superseded date) is now on #6522 (5275074980) for triage and the maintainer to weigh. I will say only this much as the dispatching seat, because it bears on evidence rather than disposition: point 2 is well-founded — the contention arm is exercised and negative, and buying more repetitions of this suite, idle or loaded, would produce nothing. What point 4 is worth is not mine to judge.

    The three contention-induced timeouts are recorded here and unclaimed. They were correctly not folded in and not filed — they belong to other packages and other lanes. @objectstack/types's looks the most tractable at 2-for-2. Whoever wants them has the signatures in §3.

    Total measurement: ≈2 h 40 m, 56 real-case executions under load plus 15,480 harness iterations. I misjudged this card's cost partway through and sent a premature stop order at 00:35Z; the maintainer corrected me and it was withdrawn. The elapsed time was the card being executed as designed, and the design was better than the one I dispatched.

    Card closed. Dev session archived. #6522's hold stands, now with the evidence its own restart condition asked for.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions