Skip to content

The tests gate reruns a native child crash once and records it - #443

Open
avifenesh wants to merge 11 commits into
mainfrom
tower-crane/T460-native-node-sigsegvs-in-gates-are-retrie
Open

avifenesh wants to merge 11 commits into
mainfrom
tower-crane/T460-native-node-sigsegvs-in-gates-are-retrie

Conversation

@avifenesh

@avifenesh avifenesh commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

The tests gate reruns a failure once when every failing test's error shows a child that ended with SIGSEGV, SIGBUS or SIGILL and empty output. T264, T412, T416, T426 and T453 each lost a rework round to these Node v26 crashes, which no JavaScript change can cause. Where apport keeps cores, the gate also reads the crashed child's core with gdb.

  • lib/native-crash.js: the parser and the core handling. The parser recognizes two forms. The first is node's child_process error object (signal: 'SIGSEGV' with every stdout/stderr/output it shows empty, plus pid), as in T412. The second is a message that ends where the child's output would follow (exited null (signal SIGSEGV): ), as in T453. A suite or parent test that failed only through failing tests inside it is judged by those tests. Spec prints no error under such a parent, and TAP marks it failureType: 'subtestsFailed'. A parent with an error of its own, such as a failed hook, still blocks the rerun.
  • lib/gates/tests.js: a head, scoped proof, reverted or host-only command whose failing tests all show a native crash reruns only the failing files once in the same checkout, through gates.tests_proof_cmd {tests}. If the rerun passes, the gate continues as if the first run had passed. If it fails, the first failure stands. The receipt records the rerun as receipt.native_rerun: {step, files, ok}.
  • lib/gate-commands.js: the evidence command check accepts the rerun's expansion. It also counts a head command the rerun rescued as passed (step: "head", ok: true, and the rerun exited 0). Without this, check tests passed while task show rejected its evidence.
  • Core capture, following the orchestrator's 2026-10-20 note: no system-wide apport settings and no /var/crash reports. On Linux, when core_pattern pipes to apport, each head, proof, reverted, host-only and rerun command runs with its soft core limit raised to the hard limit (treeExec option cores; the receipt still records the pinned command). apport writes an unpackaged binary's core to /var/lib/apport/coredump/core.EXE.UID.BOOT.PID.TIME. After each command the gate copies the core whose pid matches a crashed child into native-cores/ under the task scratch (directory 0700, file 0600) and empties the shared copy. A non-root user cannot unlink there. The gate then runs gdb -batch -nx -ex bt [node] -c CORE, records up to 40 frames as native_crash[].stack, and deletes the copy. If gdb reads no stack, core_error records why and the private copy stays. Other cores written during the command are emptied. Nothing is uploaded.
  • native_crash: [{test, signal, pid, core_file?, stack?, core_error?}] in the evidence and its check tests event. status prints native crashes: N in the last 7 days, last TASK at TIME, and its JSON always has native_crashes.
  • Docs: docs/state.md#native-crashes and the evidence and receipt paragraphs in docs/state.md and docs/cli.md. Changelog: changelog.d/T460.md.

Core analysis

No node core has been captured. Checked 2026-10-20 at 21:01 Jerusalem time, at 22:42, on 2026-10-21 at 02:10, at 05:43, and again on 2026-10-21 at 14:30:

  • /var/crash has no *node*.crash, and /var/lib/apport/coredump is empty.
  • /var/log/apport.log* (rotations from 2026-10-12 to 2026-10-21) lists 18 node SIGSEGVs, not counting this PR's deliberate fixture crashes: 7 on v26.10.0 and 11 on v26.12.0. The newest, at 2026-10-21 01:39, is tower-crane.js worktree list --json in a gate worktree running code without this change. Most are tower-crane CLI children in gate worktrees and test fixtures (tower-crane.js claim T1, task show T1, task note T1). Each ran with core limit 0, and apport logged executable does not belong to a package, ignoring, so no core or report exists for any of them. Between 02:10 and 14:30 every node SIGSEGV apport logged was one of this PR's deliberate fixture crashes (process.kill(process.pid, 'SIGSEGV')).
  • A probe in the worker sandbox with ulimit -c unlimited showed that apport 2.34 writes no core into the cwd. It logged host pid ... crashed in a container without apport support, so a sandboxed crash keeps nothing. On the host, earlier fixture crashes with limit 512 show apport choosing /var/lib/apport/coredump/core._home_..._bin_node.1000.<boot>.<pid>.<ts> and aborting only on size. That is where the gate now looks.

So no stack can be recorded yet, and no Node version is pinned. The next gate crash on the host leaves its stack in native_crash[].stack. Recording it in a Node issue draft, and pinning a version if it shows a Node or V8 bug, is the follow-up.

Verification

  • test/gates/tests.test.js "a native child crash reruns only the failing file once...": a real fixture child execs node, which kills itself with SIGSEGV. ulimit -c 1 makes apport drop the deliberate core. The fixture writes a fake apport core named with the child's pid, and fake gdb scripts stand in for gdb. The table has five rows:
    • One-off: passes. receipt.native_rerun is {step: 'head', files: ['test/crash.test.js'], ok: true}. The core is claimed into a 0700 directory, the shared copy is emptied, gdb gets the gate's node and -c CORE, stack holds its frames, and the copy is deleted.
    • One-off in a describe, and one-off in a subtest: pass.
    • Crash every time: fails with confirmed_failure and two crashes. gdb fails, so core_error is recorded and the 0600 copy stays.
    • Crash beside an ordinary failure: no rerun, and the gate fails.
    • Every row also checks that a core from before the run stays untouched and an unpaired core from the run is emptied.
  • test/gates.test.js "a one-off native child crash records native_crash...": CLI wiring. It checks the evidence, the audit event, receipt.native_rerun, the two recorded commands (exit 1, then 0), task show reporting the tests gate ok, and the status text and JSON.
  • test/native-crash.test.js: the parser variant table has an output list with content beside empty streams, and a long unclosed output list. The old single pattern over that list backtracked exponentially there (CodeQL js/redos Merge is race-free against automation: manual and automatic merges share the task reservation #118). It now splits on commas and checks each item, and the row returns in about 1 ms. The table now also has a spec suite, a spec suite with its own hook failure, a TAP subtestsFailed parent, a TAP hookFailed parent and a lone failing name. It also covers apport core_pattern detection and core selection (uid, age, emptied cores), plus the 7-day window.
  • Fails without the fix: with the aggregate clause removed, the describe row fails. With lib/gate-commands.js at its previous version, the CLI test fails with tests command receipts do not match the pinned command.
  • Run locally: test/native-crash.test.js test/gate-commands.test.js test/gate-join.test.js test/gates.test.js test/gates/tests-base.test.js test/gates/tests.test.js. All pass except gate commands run with gc.autoDetach=false..., which fails the same way at the previous head without this change. The sandbox's git shim sets 42 GIT_CONFIG_* variables.
  • npm run check:shared -- --base origin/main passes, mutants included.
  • Tests gate rework at 311acd9: the raised core limit is a shell prefix, so test/fixtures/windows-missing-command.js, which fakes cmd.exe only for the exact command tower-crane-missing-test-program, missed the prefixed command on a host where apport is core_pattern. That broke the three test/escalation.test.js Windows command-not-found tests. The prefix is now CORE_LIMIT in lib/gates/common.js, and the fixture strips it, since real cmd.exe never gets it. The three tests fail before the fix and pass after it in the sandbox, whose core_pattern is also apport.
  • Merged origin/main (host probes): plannedCommands, the receipt check in lib/gate-commands.js and both docs keep the native rerun beside receipt.host_probes.
  • Run locally after the merge: test/escalation.test.js test/native-crash.test.js test/gates/tests.test.js test/gates.test.js test/gate-join.test.js test/gates/tests-base.test.js test/gate-commands.test.js test/host-probes.test.js: 200 of 203 pass. Three fail only in the sandbox: two escalation tests find no temp directory short enough for a broker socket (/tmp is read-only here), and gc.autoDetach sees the sandbox's GIT_CONFIG_* entries. The host suite passed all three at 311acd9.
  • CodeQL rework at 2f9d0ce: test/native-crash.test.js test/gates/tests.test.js test/gates.test.js test/gates/tests-base.test.js test/gate-join.test.js after merging origin/main: 111 of 112 pass. Only gc.autoDetach fails, from the sandbox's GIT_CONFIG_* entries, and it passes when run alone with a clean environment. npm run check:shared -- --base origin/main passes.
  • Tests gate rework at 4616b5a: "past the stop line a running gate stops without a verdict and says why" in test/governor.test.js failed because kill(suitePid, 0) still answered for an exited, unreaped suite process. main has since fixed the same test (suiteStopped with waitOnRepo), so the merge at 3df5c95 takes main's version and this PR no longer changes test/governor.test.js.
  • Run locally after merging origin/main at 3df5c95: test/governor.test.js test/native-crash.test.js test/gates/tests.test.js test/gates.test.js test/gate-join.test.js test/gates/tests-base.test.js test/escalation.test.js pass, except in the sandbox's own environment: with TOWER_CRANE_STATE naming the read-only project state directory, test/gates/tests.test.js cannot take the worktree lock (EROFS), and gc.autoDetach sees the sandbox's GIT_CONFIG_* entries. With TOWER_CRANE_STATE unset, tests.test.js passes 54 of 55, and gc.autoDetach passes alone once the sandbox's GIT_CONFIG_* entries are cleared. npm run check:shared -- --base origin/main passes.

Limits

  • gdb has not run on a real core here. The sandbox blocks ptrace (gcore fails) and apport drops sandboxed crashes. The gdb call and the output parsing are covered by the fake-gdb rows. The first real crash on the host is the first real read.
  • Two gates on one host share apport's directory. A gate empties the unpaired cores written during its own command, so another gate whose child crashed in that window loses that core. It still records the crash and its rerun.
  • TAP prints no signal: for the object form, so a TAP runner only gets the line form.
  • A repeated crash still reworks, as the brief asks.

🤖 Generated with Claude Code

Pre-submit review

  • Head: 3df5c95
  • Fresh context: no
  • Acceptance: one-off child SIGSEGV reruns only the failing file and passes with a native_crash note (test/gates/tests.test.js rows one-off, one-off in a describe, one-off in a subtest; CLI wiring in test/gates.test.js "a one-off native child crash..."): pass at this head. A repeating crash fails (row every time, confirmed_failure, two crashes): pass. Captured core analyzed or absence stated with date ("Core analysis": /var/crash, /var/lib/apport/coredump and apport.log checked 2026-10-21 14:30 Jerusalem, no node core kept): pass.
  • Rework coverage: merge conflict in test/governor.test.js resolved to main's version, which fixes the same zombie-probe failure the branch had fixed; the file now matches origin/main. test/governor.test.js passes at this head.
  • Revuto at head: capped (2-round review limit, nonblocking per ci.capped_review)
  • CI at head: pending (GitHub had not yet registered the pushed head when this was written)
  • Left open: none
  • Result: pass

Note

Medium Risk
Changes tests gate pass/fail and evidence acceptance for real CI runs; core handling is scoped to private scratch but still touches host crash dumps and gdb.

Overview
The tests gate now treats one-off native child crashes (SIGSEGV/SIGBUS/SIGILL with empty output) as infrastructure, not worker failures: when every failing test matches that pattern, it reruns only the failing files once via gates.tests_proof_cmd before any base rerun, and a passing rerun lets the gate continue as if the first run succeeded.

A new lib/native-crash.js parses spec/TAP and child_process error output (including suite/parent cases), optionally raises the core limit under apport, copies matching cores into task scratch, runs gdb for a short backtrace, and records native_crash on evidence and audit events. receipt.native_rerun and lib/gate-commands.js were updated so rescued head runs still count as valid command receipts. status adds a rolling 7-day native crash count (native_crashes in JSON).

Gate lease planning includes one extra command budget per run; treeExec can prefix a POSIX core ulimit, with a Windows test fixture strip for prefixed commands.

Reviewed by Cursor Bugbot for commit 3df5c95. Configure here.

A failure whose every failing test shows a child ended by SIGSEGV, SIGBUS
or SIGILL with empty output reruns its files once in the same checkout.
A passing rerun passes the gate; every crash is recorded as native_crash
with test, signal, pid and the apport report, and status counts the last
7 days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 20, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 2df7cf1f-d1ff-4f52-b961-a96246a13cf1)

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Successful native retries produce evidence that acceptance rejects. Native crashes inside suites or subtests also fail to trigger the retry.

Comment thread lib/gates/tests.js Outdated
const again = await plan.run();
if (again.ok) {
native.notes.push(`${at} Reran ${plan.what} once in the same checkout: exit 0, so the crash is not the change's.`);
return again;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


[P2] Make recovered runs valid command evidence

The new CLI fixture runs the two-file head command unsuccessfully, then passes only the one-file retry. lib/check.js records both commands, but lib/gate-commands.js:119-122 requires a successful head command and permits only recorded head, proof or host expansions. This retry satisfies neither condition, so check tests reports success while the gate report rejects its evidence and blocks acceptance and reviewer dispatch. Record the retry selection and recovery in an audited receipt, update command validation, and extend the CLI test to check that its successful evidence counts.

Comment thread lib/native-crash.js Outdated
crashes.push({ test, ...crash });
mine++;
}
if (!mine) every = false;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


[P2] Account for aggregate failures from native-crashing descendants

Node's spec reporter includes failed suite and subtest-parent names in testFailure().names, as the existing nested integration cases in test/gates/tests-base.test.js:138-159 verify. Those aggregate parents have no crash error block after their failure header, so this sets every to false even when every failing leaf shows a native crash. Crashes inside describe() or nested subtests therefore never retry. Resolve aggregate parents through their failing descendants, as BaseTests.failingFiles does, while preserving genuine parent or hook failures.

Comment thread lib/native-crash.js Fixed
avifenesh and others added 2 commits October 20, 2026 21:19
…s are read in the gate

The tests gate records its one native crash rerun as receipt.native_rerun, and the
command check counts a head that rerun rescued. A suite or parent test that failed only
through crashed tests inside it no longer blocks the rerun. Where apport keeps cores,
gate commands run with a raised soft core limit; a crashed child's core is copied to
the task's private scratch, read with gdb into the evidence and deleted, and the
shared copy is emptied. The fixture test asserts its child's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 20, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 0c69131d-e391-429e-aeb7-edd53ea5215b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 20, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 72b17462-3082-495f-b15d-8336e617a9bc)

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Revuto completed the review and found no evidence-backed concerns.

avifenesh and others added 2 commits October 20, 2026 22:35
…mit prefix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/cli.md
#	docs/state.md
#	lib/gate-commands.js
#	lib/gates/tests.js
@cursor

cursor Bot commented Oct 20, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 739bc37f-4cba-4feb-beb3-bae725d884d8)

avifenesh and others added 2 commits October 21, 2026 02:08
… over it

CodeQL js/redos: repeated optional separators in one pattern backtracked
exponentially on a long unclosed output list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 20, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: f4fe49be-a888-4b3b-80c6-574f2fc5ac9e)

@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 336cf042-b38d-4c99-8ccb-2de8ba0e50cd)

@avifenesh

Copy link
Copy Markdown
Contributor Author

Review (Tower Crane, clean context)
SHA: d92a27a

Full review of T460 at revision 5 against its acceptance. Gate results used as supplied: tests (prove, scoped proof fails with the 8 non-test files reverted), clean and CI ok at this sha.

Acceptance:

  • One-off child SIGSEGV reruns the failing files once and passes with a native_crash note: met. lib/gates/tests.js crashRun/rerunNative rerun only the files BaseTests.failingFiles names (it returns null unless every failing name has a location, so the rerun covers every failing file). test/gates/tests.test.js rows one-off, one-off in a describe and one-off in a subtest check that the passing file is not rerun, and test/gates.test.js checks the CLI wiring: evidence, audit event, receipt.native_rerun, the two command receipts, task show counting the rescued head through the new lib/gate-commands.js rule, and status text and JSON.
  • A repeating crash fails: met (every time row, confirmed_failure, two crashes recorded). A crash beside an ordinary failure gets no rerun. A crash in the reverted run whose rerun passes makes without.ok true, so the proof fails. That direction is correct.
  • A captured core is analyzed or its absence stated with date: met. The analysis path is NativeCrash.collect/analyze (fake-gdb rows cover the stack and the core_error case). The PR body's "Core analysis" section states with dates (2026-10-20 21:01 through 2026-10-21 05:43 Jerusalem) that no node core exists, and why (apport core limit 0, unpackaged binary, sandbox pid namespace).

Scope: each file outside the named paths is needed. bin/tower-crane.js (status description), lib/render.js (status line), lib/check.js (evidence field), lib/gate-commands.js (receipt check), lib/gates/common.js (core limit), lib/agents.js (exports taskCache for the keep directory), lib/native-crash.js, docs and changelog. test/fixtures/windows-missing-command.js strips CORE_LIMIT, which the PR body explains. Receipts still record the unprefixed command, because treeExec adds the prefix after ctx.exec captured it. The PR body, changelog and both docs agree with the code.

Blocking findings

None

Blocking count: 0

Follow-ups

  • lib/native-crash.js:186 - gdb runs through child_process.execFile rather than the gate's exec helpers, so it is missing from the evidence commands and runs outside gate containment - docs/cli.md says commands lists the processes a gate ran and that gates run commands through lib/gates/common.js. Route it through the helpers (and let the receipt check skip it), or document the exception.
  • lib/native-crash.js:217 / lib/gates/tests.js:657 - each paired core gets a gdb read bounded by the full per-command test timeout, serially, and plannedCommands budgets none of it - a run with several cored crashes can outlive its gate-run lease deadline, and a joiner that cannot observe the holder would then start a second run.
  • lib/gates/tests.js:659 - a crash with no core (core_pattern not apport, sandboxed crash, line-form crash with pid null, unpaired pid) records no core field and no summary text - an explicit "no core captured" in native_crash[] or the note would answer the acceptance's absence clause in the evidence itself, not only in this PR.
  • lib/native-crash.js:209 - every same-uid core written during the command window is emptied whatever its executable - limit it to cores of node, or of the gate's process tree, so another program's or another gate's core survives. The limit is documented.

…ve-node-sigsegvs-in-gates-are-retrie

# Conflicts:
#	test/governor.test.js
@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 635ca1d5-4779-4989-a4fd-1fea49f0de5c)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants