Skip to content

fix(ci): synchronize Agent watches before starting test files - #2012

Draft
ericcaiwx-star wants to merge 2 commits into
openclaw:mainfrom
ericcaiwx-star:codex/thirty-root-07-oct10
Draft

ericcaiwx-star wants to merge 2 commits into
openclaw:mainfrom
ericcaiwx-star:codex/thirty-root-07-oct10

Conversation

@ericcaiwx-star

@ericcaiwx-star ericcaiwx-star commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Short-lived test Pods can disappear while the CI runner's kubectl observers are still connecting, so a passing file leaves no Agent activity artifact. Request Kubernetes streaming initial events and wait for both initial-state bookmarks before starting that file. Startup remains optional: a failed, missing or stalled observer stops every owned watch, removes raw streams and lets the test continue.

The fix stays in the existing capture helper and real run-tests workflow. Existing record/file caps, namespace filtering, redaction, container-log capture and result gates remain. Updates the current CI flow and contributor reference; Manual Notes preserved. No public option, Driver, dependency or runtime deployment change.

Verification

Original synchronization proof: head acb84f827e22218bdb8432c48cbb22d644503cfb, base 88024bf632672972ae7d67ceb8bfd6365d3870c7. The author follow-up below is frozen at 4b1c3280cfaaefa253010cac4176d6eb17e3e1cf.

  • Actual owned API and kubectl, pinned K3s1.35.5 image sha256:2074403abe1bded11ef3dde09d457e13be8e0b64c218b1c4f8269b4565cfbc65: unchanged7923 capture returns4ms; managed Pending Pod creation/deletion completes504ms, before the injected2.5s process-start delay; no activity retained. Current-base capture/runner blobs equal that parent. On original acb, initial synchronization takes6501ms, the actual Pod lifecycle completes6925ms, and activity is retained. Each harness exits0 asserting its expected before/after observation. Owned containers/four volumes and private kubeconfigs/streams removed. Ready-cluster state is a fixture pointing at the actual owned K3s API; this does not claim a k3d bootstrap, controller worker, running Agent, Gateway or model turn.
  • Existing run keeps bounded Agent namespace activity from passing k3d files, alone and side by side: unchanged880 parent0pass1fail0skip at missing diagnostics; original acb passes. Extended workflow also exercises a delayed initial observation, a quick passing file, a live-but-stalled observer whose failure lets the file pass, sibling streams and partial-start cleanup. Provider records in this regression are simulated; the actual API proof above is separate.
  • Original acb: official cached Node24.20 Linux, source/Git mounted read-only, no network: node --test tests/integration/ci-runner.test.mjs40pass,0fail,0skip. Final native Node26 focused activity/container-log cases2pass0skip. The exact source before commit also passed the full runner file35pass5Linux-only skips and adjacent CI preparation61pass0skip; no source edits followed those runs.
  • Full direct ESLint, TypeScript build, curated formatter, workspace boundaries, OpenAPI, specs, CI audit, docs word/site and flow checks exit0. Flow2470/reference2486words, unchanged2500hard limit. Independent frozen full-source P0–P2 review: scoped-clean,0findings,confidence0.90; reviewer did not execute code.
  • Failed preparations are separate receipts: initial native cleanup attempted to remove already-removed anonymous volumes; the next run lacked a prepared ServiceAccount; both were corrected before accepting the actual proof. Initial adjacent-test setup lacked the controller dependency graph, then61tests passed with an explicitly matched clone. Initial Linux tmpfs had noexec and rejected fixture scripts with EACCES; the disposable scratch mount was corrected, then the full40cases passed. No skipped setup or failed command is called passing evidence.

Original#582 intent and current competition/source checked. Clean virtual merge against inspected main, no covering source repair. The initial synchronization protocol is the existing Kubernetes1.35 streaming-list contract (API concepts). No Console UI change or media requirement. Original acb CI38001000471/Required114060607796 succeeded (Static114058812732). ClawSweeper input-safety screening rejected material; automated review did not run and produced no verdict. The detected value/path is withheld, so no specific hit is inferred. Draft and review gate remain unwaived pending maintainer qualification; no source/fixture change or manual re-review to evade the hold. Local independent review is not a replacement for that gate.

Failed-wait evidence follow-up

Addressed Kevin's P2 review in 4b1c3280cfaaefa253010cac4176d6eb17e3e1cf. The private container-log directory is created before optional watch startup. A failed or stalled observer is stopped and its raw streams removed; the file still receives that directory, and finish publishes its failed-wait log and snapshots. Original failure results, projection bounds and redaction remain.

  • The extended existing failed-wait workflow regression fails on unchanged acb (missing containerLogs) and passes after the repair. It asserts the failed file and passing sibling outcomes, retained log lines, Pod/Event snapshots, masking and cleanup. Linux Node 24.20, official cached digest sha256:be23f54a88d34e8824c741b19b91064094f92c1c97b194144bfc8b50d67258e2, read-only source/Git and no network: node --test tests/integration/ci-runner.test.mjs passes all 40 cases, 0 skips. Native focused cases pass 2/2, 0 skips. node --test tests/integration/ci-prepare.test.mjs tests/integration/ci-cleanup.test.mjs passes 72 cases, 0 skips.
  • Actual owned K3s 1.35.5, kubelet and kubectl through shipped run-tests and followContainerLog: old acb retains 0 structured records; repaired head retains 1 with ordinary container stdout and real Pod/Event snapshots. The same deliberately unmet Kubernetes condition still fails the test (exit 1); both proof harnesses exit 0 asserting before/after observations. An external wrapper deliberately suppresses initial BOOKMARK records from real raw watches; the Event is a benign seeded fixture. This is controlled transport proof, not a naturally observed stalled watch, k3d bootstrap, Agent/Gateway deployment or model execution. Both owned containers, their 4 volumes each and private inputs were removed.
  • Full source-context independent P0–P2 review for this follow-up reports scoped-clean, 0 findings, exit 0 (static only). Workspace, complete ESLint, TypeScript build, affected formatter, OpenAPI, docs length/site/specs, CI audit, flow and diff checks pass. The first TypeScript invocation used the wrong path for the repository alias; the actual existing .bin/tsc then passed without dependency installation. Flow/reference remain within the 2,500-word limit and Manual Notes are unchanged.

Follow-up CI Required, Static Checks and Runtime Image Fixture pass on this head. The actual GitHub tested merge 2f0d1002f4b2f8057dec7db21ef23f71e19e7249 has this head as its second parent; Checks and Conformance 2 records 1,445 passed, 0 skips, including 41 runner cases and both affected capture workflows. The local frozen source has 40 runner cases; the merged CI composition includes a main-side addition. Advisory First Agent Smoke also passed, completing at 2026-10-10T01:28:33Z. Draft and the existing input-safety qualification hold remain: no manual re-review, fixture change to evade screening, Ready transition or waived review gate.

Fixes #2011.

@clawsweeper

clawsweeper Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review blocked

Automated review did not run, so no review verdict was produced.

Reason: The input-safety check rejected material in this revision. No detected value, path, or scanner output is reproduced here.

Automatic review is on hold, including after source changes. Request a fresh re-review after resolving the failure; maintainers can also release the hold through an intentional scanner-policy update.

Next step: If this is a genuine credential, remove and rotate it. If it is an intentional test fixture, a maintainer must review and qualify it.

View the workflow run.

@ericcaiwx-star

Copy link
Copy Markdown
Contributor Author

Current head acb84f827e22218bdb8432c48cbb22d644503cfb: CI38001000471/Required114060607796 succeeds, including Static114058812732 and the actual CI-runner regressions. Local frozen Linux40pass0skip and genuine Kubernetes/kubectl before/head receipts remain in Verification; owned resources and private inputs were removed. The independent full-source review has0P0–P2findings.

ClawSweeper's input-safety check rejected material and did not run the automated review. It withheld the detected value/path, so there is no automatic verdict and no established specific cause to repair. This is a separate maintainer qualification/policy gate, not a failed product check. The changed workflow regression uses the same synthetic Pod/Event arrays as the existing test; no real credential material is introduced. That observation does not identify the withheld hit.

Keeping Draft and the review gate unwaived pending maintainer qualification. No fixture rewriting to evade screening, manual re-review, scanner change, empty commit or merge. No claim that the local independent review replaces the blocked automatic gate.

Comment thread scripts/ci/k3d-diagnostics.mjs Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI can miss short-lived Agent namespace activity before watches connect

2 participants