fix(workspace): recover ambiguous ownership without duplicate reuse

[vincentkoc]

Related: #1488

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

Fixes an issue where users reusing an SSH workspace could see a retained run abort after an acquire or renewal response was lost, even when the remote ownership mutation had succeeded.

Native Windows also needs the witnessed process tree to stop before the workspace is reused when its published supervisor record is removed or changed.

Why This Change Was Made

Acquire retries are bounded by the configured ownership wait and use same-token idempotence to reconcile a lost success response. Renewal retries are bounded by the last confirmed ownership TTL; mismatch and expiry remain terminal.

Native Windows keeps the identity-checked Job Object supervisor and revalidates its exact PID/start-time record on every poll, closing the job on record loss or mismatch. Linux, macOS, and WSL2 retain the current two-line direct-child PID/start-identity witness. This PR makes no POSIX descendant-containment claim; kernel-owned POSIX containment is tracked in #1488.

No configuration, protocol action, readiness behavior, or secret handling changes.

User Impact

Brief response loss no longer aborts a retained run while ownership is still provably within its deadline. Native Windows descendants are terminated before inspection and reuse after supervisor-record loss or tampering. POSIX completion and direct-child ownership behavior remain unchanged.

Evidence

• Exact head: bdd59ba9805b8e172fdbcccb344d8fa4b62ed997
• Base: 0364ddb5c4d2462a152db4d232f74f4f7604681b
• Focused owner and cleanup tests: pass
• Focused race: pass
• go vet ./internal/cli/...: pass
• go build -trimpath ./cmd/crabbox: pass
• Windows amd64 test-binary cross-compile: pass
• gofmt and git diff --check: pass
• Hosted exact-head CI: all required checks pass
• Exact-head ClawSweeper re-review: reviewed bdd59ba9805b; no source-code finding, but proof remains blocked on the native Windows authority boundary and a complete WSL2 lifecycle
• Fresh POSIX SSH proof: pass. The retained workload acquired ownership, ran for 17 seconds across renewal, preserved stdout/stderr markers, exited 0, released the owner and lease, and left owner=0 child=0 run=0 launcher=0 tmp=0 with zero local spools.
• Fresh native Windows proof: infrastructure-blocked before the changed Job Object supervisor path. Direct Windows OpenSSH remained available, but owner renewal/staging failed closed before the workload. Exact task processes and eight private stdin files were mapped and removed; final owner=0 child=0 run=0 witness=0 tmp=0, zero local spools, lease released. The earlier combined-head proof covered record removal/tampering and descendant termination, but it is not substituted for fresh exact-head final-effect proof.
• Fresh Windows-to-WSL2 proof: infrastructure-blocked before workload execution. Warmup completed; direct Windows OpenSSH and direct wsl.exe controls both passed. The exact owner acquire stayed ambiguous for 168 seconds while broker heartbeats timed out, then failed closed. Three exact task wrappers were mapped and stopped; final owner=0 child=0 run=0 launcher=0 tmp=0, zero local spools, lease released and absent.
• Production implementation: +158/-40, net +118 lines, justified by bounded reconciliation and the native Windows security boundary
• Tests: +467/-5; docs and changelog: +11/-3

Current Blocker

Do not merge until a maintainer resolves ClawSweeper's exact-head proof hold. Fresh native Windows record-tamper final-effect proof and a successful Windows-to-WSL2 acquire/renew/workload/release lifecycle remain unavailable; all exact task leases, processes, remote residue, and local spools are cleaned.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 24, 2026, 3:49 AM ET / 07:49 UTC.

ClawSweeper review

What this changes

This PR retries ambiguous SSH workspace-owner acquire and renewal responses within bounded deadlines, adds a native-Windows Job Object supervisor for witnessed workloads, and updates the related tests and run documentation.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 5 items remain

Keep open. Static review found no discrete patch defect, but the PR’s own exact-head evidence still lacks the required native-Windows final-effect and complete WSL2 lifecycle proof for this authority-bearing change.

Priority: P1
Reviewed head: bdd59ba9805b8e172fdbcccb344d8fa4b62ed997

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The implementation has focused static coverage, but the required authority-boundary behavior remains unproven on the exact head.
Proof confidence	🦪 silver shellfish (2/6)	Needs stronger real behavior proof before merge: Authority-chain proof required: the supplied material has exact-head POSIX success, but its fresh native-Windows and Windows-to-WSL2 attempts failed before the changed workload paths could demonstrate final-effect recovery and cleanup. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🦐 gold shrimp (3/6)	Security review found an item that needs attention.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: Authority-chain proof required: the supplied material has exact-head POSIX success, but its fresh native-Windows and Windows-to-WSL2 attempts failed before the changed workload paths could demonstrate final-effect recovery and cleanup. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	7 items	Bounded acquire and renewal reconciliation: The PR retries an ambiguous acquire through the configured wait and retries renewal only until the prior confirmed TTL deadline; mismatch and expiry remain terminal. Native Windows final-effect implementation: The supervisor attaches the identity-checked workload process to a kill-on-close Job Object, publishes its own PID/start identity, and closes the job when the record no longer matches. Focused regression coverage: Tests cover lost acquire/renew responses and, on native Windows, child-record removal and identity tampering terminating both workload leader and descendant before reuse.
Findings	None	None.
Security	Needs attention	Prove record invalidation reaches the final side effect: The Job Object is intended to kill the workload when the supervisor record is removed or changed, but the supplied exact-head live evidence did not reach that path; source tests alone do not establish the production final effect.

Live Verification

Command: go run ./cmd/crabbox run --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

Assertions:

• FAIL expect_output: Usage:

How this fits together

The workspace-owner subsystem serializes access to reused SSH workspaces before sync, execution, collection, and cleanup. It turns remote ownership responses and child-record state into either permission to continue a retained run or a fail-closed cleanup/reuse decision.

flowchart LR
  A[CLI run request] --> B[Workspace owner protocol]
  B --> C[SSH owner and child records]
  C --> D{Response and record valid?}
  D -->|confirmed| E[Retained workload]
  D -->|ambiguous response| F[Bounded reconciliation]
  F --> E
  D -->|record changed| G[Stop native Windows job]
  G --> H[Safe workspace reuse]

Loading

Before merge

• Add real behavior proof - Needs stronger real behavior proof before merge: Authority-chain proof required: the supplied material has exact-head POSIX success, but its fresh native-Windows and Windows-to-WSL2 attempts failed before the changed workload paths could demonstrate final-effect recovery and cleanup. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Resolve security concern: Prove record invalidation reaches the final side effect - The Job Object is intended to kill the workload when the supervisor record is removed or changed, but the supplied exact-head live evidence did not reach that path; source tests alone do not establish the production final effect.
• Resolve merge risk (P1) - The new native-Windows record-to-Job-Object authority boundary has no exact-head live proof that record removal or reassignment kills the leader and descendant before inspect or reuse.
• Resolve merge risk (P1) - A successful exact-head Windows-to-WSL2 acquire, renew, workload, and release lifecycle is still absent.
• Resolve merge risk (P1) - The current default-branch source could not be fetched locally for an independent divergence comparison.

Findings

• [medium] Prove record invalidation reaches the final side effect — internal/cli/workspace_owner.go:1117

Agent review details

Security

Needs attention: The source contains record revalidation and kill-on-close handling, but the changed authority boundary still needs exact-head final-effect proof.

Review metrics

Metric	Value	Why it matters
Production versus regression coverage	production +158/-40; tests +467/-5; docs/changelog +11/-3	The authority and recovery change is concentrated in one production file but has substantial focused regression coverage across five changed files.

Merge-risk options

Maintainer options:

1. Prove the final authority effects (recommended)
   Before merge, post redacted exact-head evidence that record removal and identity tampering stop the native-Windows workload tree before inspection or reuse, plus a complete WSL2 lifecycle.
2. Pause for a viable proof environment
   Keep this PR open without merging until an environment can exercise the required native-Windows and WSL2 paths end to end.

Technical review

Best possible solution:

Add redacted exact-head native-Windows record-tamper final-effect evidence and a successful Windows-to-WSL2 retained-workload lifecycle before merging.

Do we have a high-confidence way to reproduce the issue?

Yes for the central source path: the pre-PR acquire code returned exit 7 immediately on a transport error, and the new focused tests simulate lost acquire and renewal responses. No complete exact-head native-Windows or WSL2 live reproduction is available.

Is this the best way to solve the issue?

Unclear; bounded same-token reconciliation and identity-checked Windows supervision are narrow approaches, but their required final effects remain unproven on the exact head.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a131bae570da.

Labels

Label changes:

• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
• remove rating: 🦐 gold shrimp: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

• P1: Workspace-owner recovery affects retained SSH runs and can abort or incorrectly reuse active user workspaces.
• merge-risk: 🚨 security-boundary: The PR changes the authority record that decides whether a native-Windows workload tree is terminated before workspace reuse.
• merge-risk: 🚨 availability: Bounded acquire and renewal reconciliation directly controls whether retained runs continue or fail closed after response loss.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Authority-chain proof required: the supplied material has exact-head POSIX success, but its fresh native-Windows and Windows-to-WSL2 attempts failed before the changed workload paths could demonstrate final-effect recovery and cleanup. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

Security concerns:

• [medium] Prove record invalidation reaches the final side effect — internal/cli/workspace_owner.go:1117
  The Job Object is intended to kill the workload when the supervisor record is removed or changed, but the supplied exact-head live evidence did not reach that path; source tests alone do not establish the production final effect.
  Confidence: 0.96

What I checked:

• Bounded acquire and renewal reconciliation: The PR retries an ambiguous acquire through the configured wait and retries renewal only until the prior confirmed TTL deadline; mismatch and expiry remain terminal. (internal/cli/workspace_owner.go:294, bdd59ba9805b)
• Native Windows final-effect implementation: The supervisor attaches the identity-checked workload process to a kill-on-close Job Object, publishes its own PID/start identity, and closes the job when the record no longer matches. (internal/cli/workspace_owner.go:1081, bdd59ba9805b)
• Focused regression coverage: Tests cover lost acquire/renew responses and, on native Windows, child-record removal and identity tampering terminating both workload leader and descendant before reuse. (internal/cli/workspace_owner_test.go:352, bdd59ba9805b)
• Workspace-owner history: Local history attributes the current workspace-owner line to repeated merged work by Vincent Koc, with earlier renewal-quiescence and Windows transport changes by Peter Steinberger. (internal/cli/workspace_owner.go:294, bdd59ba9805b)
• Proof remains incomplete: The PR body reports successful exact-head POSIX behavior but says fresh native-Windows and Windows-to-WSL2 attempts stopped before the changed workload paths, so neither proves the required final effects. (bdd59ba9805b)
• Current-main comparison unavailable locally: The checkout has the PR head and its base, but the supplied current-main object requires a promisor fetch that fails because network DNS is unavailable; no claim of an independent current-main source comparison is made. (a131bae570da)

Likely related people:

• vincentkoc: Current local history shows several merged workspace-owner and WSL2 changes by this contributor before the reviewed head. (role: recent workspace-owner contributor; confidence: high; commits: bdd59ba9805b, 0364ddb5c4d2, 0cf9303a2c32; files: internal/cli/workspace_owner.go, internal/cli/workspace_owner_test.go)
• Peter Steinberger: Local history associates earlier workspace-owner renewal-quiescence and native-Windows transport work with this contributor. (role: earlier workspace-owner contributor; confidence: medium; commits: e6dd2151171b, d4c5c77ce331; files: internal/cli/workspace_owner.go, internal/cli/run.go)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Add final-effect proof that the nearest forbidden case—removing or changing the supervisor record—closes the Job Object and terminates the workload tree before inspect or reuse.
• Add a redacted exact-head Windows-to-WSL2 acquire, renew, workload, release, and zero-residue transcript.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (19 earlier review cycles; latest 8 shown)

• reviewed 2026-08-21T00:14:44.269Z sha 94d445d :: needs maintainer review before merge. :: none
• reviewed 2026-08-23T21:02:36.903Z sha 791aaa7 :: needs changes before merge. :: [P2] Use release-neutral legacy-record wording
• reviewed 2026-08-23T21:26:35.168Z sha 8c0bcb2 :: needs real behavior proof before merge. :: [P1] Expire the Windows job when its child record changes
• reviewed 2026-08-23T21:56:48.111Z sha b7c62e4 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-23T22:05:14.385Z sha b7c62e4 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-23T22:29:28.118Z sha d08073d :: needs real behavior proof before merge. :: none
• reviewed 2026-08-23T22:50:21.528Z sha d08073d :: needs real behavior proof before merge. :: none
• reviewed 2026-08-24T07:08:10.386Z sha bdd59ba :: needs real behavior proof before merge. :: none

[vincentkoc]

@clawsweeper re-review

The reviewed head is stale; f92135ad0a4cbf2e1d36eb316dcfae0d5695528a applies the publication, overlapping-witness, Windows PID-reuse, and normal-descendant fixes.

The legacy two-line POSIX record remains intentionally fail-closed. It has no sentinel identity or process-group authority, so it cannot prove descendant ownership or safe termination. Treating it as recoverable would reintroduce the unrelated-process/PID-reuse risk this change removes. This is the explicit owner decision for this persisted-state boundary; no compatibility path is intended.

Please review the exact head.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/32399330925
• Updated: 2026-08-20T17:45:27.015Z

[vincentkoc]

@clawsweeper re-review

Please review exact head eda596dc3c1a8347b9c08bed7da00d7b5fb316c2.

This head closes the two startup-cleanup findings from Sol/high: POSIX no longer reopens an orphaned FIFO during failed startup, and native Windows establishes the identity-bound job supervisor before child-record publication and cleans every rejected startup through exact process handles. The production delta remains at the approved net +160 cap.

The explicit legacy-record decision remains unchanged: old two-line POSIX records fail closed because they cannot prove sentinel or process-group authority.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/32400871089
• Updated: 2026-08-20T18:02:03.274Z

[vincentkoc]

@clawsweeper re-review

Please review exact head b7c62e4bbe8fc485ce41001157c47dec431f71f6.

This fixes the prior security P1 at the native Windows supervisor owner: after a publication handshake, the supervisor revalidates its exact PID/start-time child record on every poll and closes the kill-on-close job on record removal or mismatch. A Windows-only regression proves both record removal and identity tampering terminate the workload leader and descendant before inspect/reuse, then release with zero residue.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

[vincentkoc]

@clawsweeper re-review

Exact head: d08073d60dc78b23b88939bd17b946aa66ed124a

Please review the scope-B boundary: bounded ambiguous acquire/renew reconciliation, unchanged POSIX two-line direct-child witnessing, and native Windows-only Job Object containment with supervisor-record revalidation. Kernel-owned POSIX descendant containment is deferred to #1488.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/32671522907
• Updated: 2026-08-23T22:45:01.223Z

[vincentkoc]

Exact-head live proof update for d08073d60dc78b23b88939bd17b946aa66ed124a:

• Native Windows inbox OpenSSH: acquire -> renew -> inspect -> release, command completion ok, residue 0.
• Native Windows final effect: both child-record removal and identity tampering closed the Job Object; workload leader and descendant exited before reuse; residue 0.
• POSIX SSH: acquire -> renew -> inspect -> release, command completion ok, renewal observed after more than 10 seconds, residue 0.
• WSL2 ownership: acquire and renewal succeeded, but the standard witnessed workload command exited before completion/inspect. The generated argv-backed WSL invocation measures 18,870 bytes against Windows' 8,191-byte boundary.
• Diagnostic control: the existing exact-byte stdin-staged WSL control-script runner completed acquire -> renew -> inspect -> release with residue 0, confirming that stdin staging is the working transport. That control path is not the standard runSSHStreamResult workload path, so I am not treating it as operator-path proof.
• Cleanup: both exact cloud leases released; Windows and Linux inventories 0; claims, remote fixtures, and local proof artifacts removed.

Hosted exact-head CI is fully green (20 successful, 2 skipped, 0 pending/failing). This is a code blocker, not an infrastructure blocker. The PR remains unmerged pending a reviewed transport repair and fresh WSL2 proof.

[vincentkoc]

@clawsweeper re-review

Exact head remains d08073d60dc78b23b88939bd17b946aa66ed124a.

The PR body and #1442 (comment) now contain redacted exact-head live evidence. Native Windows and POSIX pass with zero residue, but the real WSL2 workload path is blocked: its ownership-expanded argv wrapper is 18,870 bytes against the Windows 8,191-byte limit. Please review this as a concrete transport blocker; do not clear the proof or merge-risk labels based on the stdin-staged control-path diagnostic.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

[vincentkoc]

@clawsweeper re-review

Rebased onto the landed WSL2 stdin transport prerequisite. Please review exact head bdd59ba9.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.