Record isolated kqueue deletion-flush evidence for #83

Preserve complete red and green macOS job logs, checkout provenance, dependency source hashes and source-based attribution. Archive read-back comparisons passed. Independent review accepted attribution with bounded scope; parent corrected its inverted worker race condition and recovered the complete baseline checkout log. One green run is not determinism; write skips, dependency pin, separate EBADF probe and issue status remain unchanged. Parent just lint and just ci passed.

Author: mattrobenolt

PRODUCTION_READINESS.md

@@ -925,9 +925,11 @@ backend, and single-backend coverage is not backend coverage.
 
 **Gaps (tracked under #76's successors):** in-flight cancellation is now covered
 for both reads and writes, abortive and orderly, on io_uring and epoll (#83).
-kqueue remains the unproven island: the abortive read test stalls there for
-reasons still unestablished, and both write variants share the same
-cancel-then-close shape and are skipped for the same reason. The epoll fix works
+kqueue cancellation remains unproven in the shipped dependency: the abortive
+read stall is attributed to libxev's dropped `EV_DELETE` during `tick(0)` by the
+[isolated deletion-flush experiment](docs/research/XEV_KQUEUE_83/20260913-flush/README.md).
+The fix is not adopted. Both write variants remain skipped and were not
+validated by that experiment; #83 stays open. The epoll fix works
 around what looks like an upstream libxev defect: its epoll TCP watcher
 duplicates the fd per operation and the normal completion path closes that
 duplicate, but the cancellation path (`stop_completion`) only does
@@ -970,7 +972,14 @@ after half a record would be un-authenticatable garbage.
 The abortive in-flight-read test stalls on macOS/kqueue: after the server's
 cancellation completes, the client's plain socket close is armed
 (`phase = .released`) but its callback never fires and `Loop.active == 0`.
-Unresolved, and now bounded by evidence rather than by hypothesis.
+CI run `34742256947` reproduces this exact state at diagnostic revision
+`f3d0a0e`; run `34742815211` passes the same enabled test at `7715eb5` with only
+upstream mitchellh/libxev#224's deletion-flush hunk applied to the pinned source.
+The tick-local deletion is otherwise discarded on `wait == 0`. A stale EOF
+can re-fire and decrement `active` again, preventing the loop from reaching
+thread-pool completion migration. Source analysis and the isolated red/green
+experiment establish the abortive-read attribution, not deterministic behavior
+across all interleavings. The production pin and kqueue skips are unchanged.
 
 Two hypotheses were tested against real macOS runs and both are dead: libxev
 mis-accounting `active` on the cancel path (a single-socket probe comes back
@@ -986,10 +995,10 @@ other has a read armed:
       src/backend/kqueue.zig:1285 in perform   (xev_posix.close(op.fd))
       src/backend/kqueue.zig:960 in thread_perform
 
-That is a double close of an already-closed fd in the thread-pool worker.
-Whether it shares a root cause with the `Conn` stall is unestablished: the ztls
-test stalls rather than panicking, so they may be separate defects in the same
-area. Upstream-reportable either way.
+The thread-pool worker attempted to close an invalid fd. The earlier closer or
+possible duplicate scheduling remains unestablished. The deletion-flush
+experiment did not run this probe and does not establish its relationship to
+the `Conn` stall.
 
 **macOS is now CI-gated.** A `macos-15` job runs `just integrations-ci` on kqueue,
 scoped to the 0.16 integrations rather than the whole lane (conformance needs a
@@ -1000,15 +1009,14 @@ manual run — the 21/21 that closed #76 was stale within a day.
 
 The runner immediately narrowed #83, which is the argument for having it. Only
 the **abortive** close with a read in flight stalls on kqueue; the **orderly**
-close passes there and runs normally. The two differ by exactly one thing — the
-close_notify write between the cancel and the socket close — so an extra
-completion cycle at that point is enough to unstick it. That is the sharpest clue
-available and was invisible from a single manual run, which had only ever
-exercised the abortive case.
-
-The abortive case is skipped on kqueue rather than asserted-as-failing: a stalled
-loop's teardown crashes, so the failure cannot be caught and asserted. In-flight
-write cancellation remains separately unproven.
+close passes there and runs normally. The extra close_notify write changes the
+event path, but timing alone was not accepted as proof of a repair. The isolated
+deletion-flush experiment preserves the abortive test's trigger and timing.
+
+The abortive case remains skipped on kqueue until a dependency fix is adopted.
+Earlier stalled-loop teardowns crashed; the preserved CI baseline instead
+returns `LoopStalled` cleanly. In-flight write cancellation remains separately
+unproven.
 
 
 

docs/research/XEV_KQUEUE_83/20260913-flush/README.md

@@ -0,0 +1,80 @@
+# Kqueue deletion-flush experiment (#83)
+
+Project status and remaining scope live in
+[PRODUCTION_READINESS.md](../../../../PRODUCTION_READINESS.md).
+This directory preserves one baseline and one candidate macOS CI job log.
+`SHA256SUMS` covers the losslessly compressed logs; both decompressed files were
+compared byte-for-byte with the original downloads.
+
+## Provenance
+
+Both jobs use `macos-15`, Zig 0.16.0, and `just integrations-ci`.
+The common main revision is `ce434db9b29a1fc51034de66241d9ed0c2e4f6b0`.
+GitHub tests synthetic merge commits; each complete job log includes its
+checkout line, not merely the test output.
+
+- Baseline: [run 34742256947](https://github.com/mattrobenolt/ztls/actions/runs/34742256947),
+  job `103683823687`, branch revision
+  `f3d0a0e67e1c8027a5b382be44da003453c887af`, merge checkout `5583296`.
+- Candidate: [run 34742815211](https://github.com/mattrobenolt/ztls/actions/runs/34742815211),
+  job `103685296932`, branch revision
+  `7715eb57c9f81711ccf687eabc8cee759624773a`, merge checkout `85bd368`.
+- Diagnostic branch: [PR #97](https://github.com/mattrobenolt/ztls/pull/97).
+  The baseline changes one line: it enables the existing abortive-read test on
+  kqueue. The candidate adds a macOS-only preparation step and patch files.
+  Neither changes the test timing, assertions, callbacks, or write skips.
+- libxev pin: `9ce8e8e6ff89e583258a7f8e7adeeeaeae8611bf`, package
+  `libxev-0.0.0-86vtcwIRFADbH4hk-EjROXxlrKIRPQdA41XiTSytYO-F`.
+  The candidate modifies only its checkout-local `src/backend/kqueue.zig`:
+
+  ```diff
+  -            if (wait == 0) break;
+  +            if (wait == 0 and changes == 0) break;
+  ```
+
+  This is the deletion-flush hunk from
+  [mitchellh/libxev#224](https://github.com/mitchellh/libxev/pull/224),
+  inspected at `7497c85dfc3ae5141308944f2cd47c836772c300`.
+  No other hunk is applied. The script verifies the pristine file hash before
+  applying the patch, then prints the changed hash before the integration gate:
+
+  - Before: `01c0c18f47f81b718f03c4d15279c88852ad148a423fa08d5b3f128cfd9db069`
+  - After: `de8b5fcdec908f9e0766bb711ad26d81dce18d7deaca26d8d84740e7ada948bc`
+
+The cache mutation is an isolated diagnostic technique, not a production
+installation method. No dependency override or CI patch is adopted on main.
+
+## Observations
+
+The baseline's enabled abortive-read test returns `error.LoopStalled`:
+
+```text
+stalled on kqueue: srv=.{ .state = .closed, .wait = .idle, .phase = .released }
+  cli=closing/idle/released active=0 events={ .read_canceled, .closed }
+```
+
+The candidate executes that same test and reports `PASS`. The orderly-read test
+also passes in both logs. The two write-test `PASS` lines are empty conditional
+bodies on kqueue: they are not evidence of write cancellation coverage.
+
+## Source analysis and limits
+
+In pinned libxev `Loop.tick`, `changes` and the deletion buffer are local.
+The kevent event path stages a deletion after a `.disarm` callback, then the
+unconditional `wait == 0` break discards it. A level-triggered EOF can therefore
+re-fire for the retired read. Unlike the completions-queue path, the event path
+unconditionally decrements `active` after `.disarm`.
+
+The client read callback requests its thread-pool socket close. If the stale
+read event is reaped before the worker closes the fd, that extra decrement can
+consume the close operation's active count. At zero, the loop gate excludes
+entry to the thread-pool completion migration, so the close callback remains
+undelivered. This explains the observed state without a missing cancel.
+The changed condition keeps the tick alive to submit its staged deletion.
+
+The source analysis and isolated red/green experiment support attribution of
+this abortive-read stall to the dropped deletion. The logs are not per-event
+traces; they do not directly record each intermediate callback or worker
+interleaving. One green run does not establish deterministic behavior or validate
+all upstream PR hunks. The separate `two_conn_cancel` EBADF probe was not run
+in this experiment, and its relationship to this stall remains unestablished.

docs/research/XEV_KQUEUE_83/20260913-flush/SHA256SUMS

@@ -0,0 +1,2 @@
+5fd1e64cd53e5c147032f7473d7b73801720878e9c450e5b67bbff8c4d7dbd5b  baseline-34742256947.log.xz
+24933169829b0d2e6f29a0bf7d9a36a767abbd5d176423c90ab297ec5a47bb66  candidate-34742815211.log.xz