Event feed connector: the run loop and tier-2 driver (2/3) - #705
Conversation
There was a problem hiding this comment.
Pull request overview
Adds the experimental Go Event Feed reference connector, deterministic conformance infrastructure, WebSocket transport, and checkpoint persistence.
Changes:
- Implements the push/poll state machine, recovery, deduplication, and checkpointing.
- Adds real and fake transports plus tier-2/tier-3 conformance tests.
- Documents the experimental API and architecture exception.
Tip
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.
Reviewed changes
Copilot reviewed 54 out of 55 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
AGENTS.md |
Registers Event Feed infrastructure. |
CONTRIBUTING.md |
Documents conformance verification. |
go/README.md |
Adds Event Feed usage guidance. |
go/go.mod |
Adds WebSocket dependency. |
go/go.sum |
Locks WebSocket dependency. |
eventfeed/backoff.go |
Implements retry timing. |
eventfeed/backoff_test.go |
Tests retry timing. |
eventfeed/cable.go |
Implements cable framing. |
eventfeed/cable_test.go |
Tests cable framing. |
eventfeed/catchup.go |
Implements catch-up and streaming. |
eventfeed/catchup_test.go |
Tests catch-up behavior. |
eventfeed/checkpoint.go |
Defines checkpoint contracts. |
eventfeed/clock.go |
Implements production timers. |
eventfeed/clock_test.go |
Tests production timers. |
eventfeed/connector.go |
Defines the public connector. |
eventfeed/connector_test.go |
Tests connector construction. |
eventfeed/continuation.go |
Validates continuation URLs. |
eventfeed/dedupe.go |
Implements delivered-ID deduplication. |
eventfeed/dedupe_test.go |
Tests deduplication. |
eventfeed/digest.go |
Implements filter digests. |
eventfeed/digest_test.go |
Tests shared digest vectors. |
eventfeed/doc.go |
Documents the package contract. |
eventfeed/errors.go |
Defines terminal errors. |
eventfeed/errors_test.go |
Tests error taxonomy. |
eventfeed/event.go |
Defines feed events. |
eventfeed/event_test.go |
Tests event decoding. |
eventfeed/export_test.go |
Exposes test-only hooks. |
eventfeed/filestore.go |
Implements file checkpoints. |
eventfeed/filestore_test.go |
Tests file checkpoints. |
eventfeed/filters.go |
Defines filter validation. |
eventfeed/filters_test.go |
Tests filter validation. |
eventfeed/loop.go |
Implements connector lifecycle. |
eventfeed/loop_test.go |
Tests lifecycle behavior. |
eventfeed/reconnect_test.go |
Tests reconnect and staleness. |
eventfeed/recovery.go |
Implements poll recovery. |
eventfeed/recovery_test.go |
Tests recovery paths. |
eventfeed/scenario_conformance_test.go |
Replays conformance fixtures. |
eventfeed/scenario_fixture_test.go |
Decodes fixture contracts. |
eventfeed/scenario_harness_test.go |
Provides scenario harnessing. |
eventfeed/scenario_selftest_test.go |
Tests driver strictness. |
eventfeed/seams.go |
Defines connector seams. |
eventfeed/transport.go |
Enforces cable URL policy. |
eventfeed/transport_contract_test.go |
Defines transport contract tests. |
eventfeed/transport_test.go |
Tests cable URL policy. |
eventfeed/websocket_transport.go |
Implements WebSocket transport. |
eventfeed/websocket_transport_test.go |
Tests real WebSocket behavior. |
eventfeed/feedtest/clock.go |
Adds deterministic virtual time. |
eventfeed/feedtest/clock_test.go |
Tests virtual time. |
eventfeed/feedtest/minter.go |
Adds scripted ticket minting. |
eventfeed/feedtest/minter_test.go |
Tests scripted minting. |
eventfeed/feedtest/polls.go |
Adds scripted polling. |
eventfeed/feedtest/polls_test.go |
Tests scripted polling. |
eventfeed/feedtest/store.go |
Adds scripted checkpoints. |
eventfeed/feedtest/transport.go |
Adds scripted cable transport. |
eventfeed/feedtest/transport_test.go |
Tests scripted transport. |
Suppressed comments (1)
go/pkg/basecamp/eventfeed/filestore.go:230
- The rename is atomic but not durable without syncing the staged file and parent directory. After a system crash, the first checkpoint file can disappear; the next
Loadthen reports Missing and starts at the present, which can skip history rather than merely replay from an older position. SinceSaveand the package advertise durable checkpointing, sync the file before rename and the directory after rename, or stop claiming crash durability and avoid treating disappearance as a safe present entry.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ceb8398f4c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.
Suppressed comments (4)
go/pkg/basecamp/eventfeed/websocket_transport.go:73
- The offered subprotocol is not verified after the handshake. coder/websocket v1.8.15 accepts a 101 response with an empty
Sec-WebSocket-Protocol, so this can return a connection even thoughactioncable-v1-jsonwas never negotiated, contrary to theCableTransportcontract. Checkconn.Subprotocol()and reject/close a missing selection; add a server case that intentionally selects none.
go/pkg/basecamp/eventfeed/catchup.go:302 - This parks an already-observed overflow until the poll returns. SPEC §23 requires semantic signals at the first consumer-context opportunity after the condition arises (with “before the next save” only as the outer bound), so a stalled poll can postpone the handler forever even though this goroutine has received the dropping frame. Dispatch the overflow immediately here; an Accept disposition can continue awaiting the poll, while Terminate should cancel the attempt/poll.
} else if over {
// The buffer, not the socket: the call is unaffected and is
// awaited to completion, and the drop's disposition runs
// before this page's position moves anything durable.
l.deferred = &deferredFrame{item: item, overflow: true}
if l.hooks.frameDeferred != nil {
l.hooks.frameDeferred(true)
}
r := <-done
return r.page, false, r.err
go/pkg/basecamp/eventfeed/websocket_transport.go:73
checkCableURLexplicitly accepts case-insensitive schemes (the new unit test includesWSS://), but this passes that original spelling to coder/websocket. In v1.8.15 its handshake switch recognizes only lowercasews/wss, so a URL accepted by policy fails as a transient dial and is retried indefinitely. Normalize only the scheme before dialing, while leaving the ticket-bearing remainder unchanged.
This issue also appears on line 70 of the same file.
go/pkg/basecamp/eventfeed/websocket_transport.go:183
- This synchronous graceful close can block teardown for several seconds: coder/websocket v1.8.15 waits up to 5 seconds to write the close frame and another 5 seconds for the peer response. Because
disposecalls this before cancelling the attempt, caller cancellation,Connector.Close, terminal outcomes, and reconnects can all stall on a live peer that ignores the handshake. Use a bounded teardown strategy that preserves the required close frame without letting the library's full handshake timeout delay the universal Closed edge.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 55 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
go/pkg/basecamp/eventfeed/catchup.go:504
- A live event admitted by
drainScanis stranded when the buffer was empty at the start of this iteration:batchstays empty, so this returns even though the scan just repopulatedl.buffer. Streaming never drains that buffer, delaying the event until a later repair walk and allowingcaught_up(and a held save) to happen first. Continue the loop when the scan added events.
batch := l.buffer.take()
if out, done := l.drainScan(at, &budget); done {
return out, true
}
if len(batch) == 0 {
return cycleOutcome{}, false
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6eb58e6010
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Round-2 note on the suppressed comments, since three of them are load-bearing and one changed the design. catchup.go:302 (overflow parked until the poll returns) — taken, and it reframed the visible finding above it. Rather than add a third dispatch site I deleted the overflow deferral outright; catchup.go:504 (live event stranded by drainScan) — real, and a defect introduced by my own round-1 change: websocket_transport.go:73 ×2 and :183 — all three taken and in progress: the negotiated subprotocol is never verified after the handshake, a Thanks for putting the sharp ones in the suppressed block; they've been the most useful part of both rounds. |
|
Closing out the remaining suppressed comments from round 2 (02fec5f). Two fixed, one declined with evidence. Unverified subprotocol — real, fixed. Confirmed in the library rather than assumed: Unbounded graceful close — real, fixed. Case-insensitive scheme — not a defect, declined. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.
Suppressed comments (3)
go/pkg/basecamp/eventfeed/catchup.go:158
- A socket outcome deferred while this poll was in flight is skipped when the poll itself fails. For terminal poll branches,
recoverPollcallsdisposeAttempt, which clearsl.deferred; for retryable failures, the deferred frame can remain undispatched through arbitrarily many retries. In particular, aninvalid_event_stream_commandobserved during CatchingUp can be replaced bypoll_failed/authorization_failedor delayed indefinitely, despite SPEC §23 requiring protocol-fatal to terminate from every socket-open state. Dispatch the already-observed socket outcome here before applying poll recovery; a failed poll has no successful page boundary left to finish.
if p.err != nil {
step, out, done := l.recoverPoll(at, cursor, p.err)
go/pkg/basecamp/eventfeed/transport.go:40
- Checking
u.Hostdoes not ensure that the URL has a hostname. For example,wss://:443/cablehasHost == ":443"but an emptyHostname(), so it passes the policy check and is classified as a transient dial failure instead of terminalinvalid_cable_url, causing repeated re-mints/dials for a structurally unusable URL.
go/pkg/basecamp/eventfeed/cable.go:250 - The tier-2 push-event schema requires all nine keys, including presence-bearing
visible_to_clients, and SPEC §23 treats a correlated message missing a required event key as an invalid frame. Omitting this field from the presence checks accepts both an absent value and JSONnull, exposing a nil value on a push event instead of taking the socket-failure recovery edge.
{"creator_id", p.CreatorID != nil},
{"recording_id", p.RecordingID != nil},
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02fec5ff1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.
Suppressed comments (1)
go/pkg/basecamp/eventfeed/catchup.go:160
- A deferred socket outcome is skipped when the in-flight poll returns an error.
recoverPollmay retry, increment authorization failures, or terminate, and disposal then clearsl.deferred; this can even swallow an already-observedinvalid_event_stream_commandinstead of producingprotocol_fatal. Since no page succeeded, dispatch the deferred outcome before classifying the poll error (the finish-page ordering only applies to successful pages).
if p.err != nil {
step, out, done := l.recoverPoll(at, cursor, p.err)
if done {
return out, "", true
Holding two findings open deliberately: the in-flight-poll mechanism has drawn four roundsTwo round-4 findings — the suspendable bound in The ledger on one mechanism, in order:
Each fix was principled and each was verified, but four rounds of edge-findings on one structure is evidence about the structure. What they all orbit: the state machine both awaits a poll seam call and services the frame queue during it, parking one out-of-band frame in a single The concurrency is not optional — fixtures 01 and 19 require a live frame admitted after
I lean (a) plus a note, because the deferral that remains is the one SPEC explicitly sanctions (transition 21's deferred consumption) and (b) risks the delivery-ordering guarantees that nine fixtures and the save-ordering invariant pin. But it is a judgment call about this PR's core, so it goes to a human rather than to me. The other three round-4 findings (synchronous cancellation on |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9103dd83a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 55 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
go/pkg/basecamp/eventfeed/loop.go:152
ctxis the attempt context, which is a child ofrunCtx.Connector.Close(or caller cancellation) therefore cancels the pump immediately, before the state machine can reachdisposeand callconn.Close. The default WebSocket read may abort the socket on that cancellation, so normal shutdown can still produce the abrupt disconnect that the close-before-cancel ordering is meant to prevent. Give the pump a cancellation scope that is canceled only by disposal after the socket close; HTTP seam calls can continue using the attempt context.
go/README.md:615- This says the connector performs no wire I/O, but the package directly performs the sanctioned WebSocket cable dial. That contradicts both the implementation and the architecture rule. Limit the claim to HTTP requests and explicitly name the cable dial exception.
**Experimental: the Layer-1 seam adapters have not landed yet.** The connector performs
no wire I/O of its own — every HTTP exchange reaches the wire through a seam backed by
a generated operation — and the adapters that build those seams over the generated
`CreateStreamTicket` and `PollEvents` operations are still to come. Until they do, a
consumer must supply the `TicketMinter` and `PollSource` implementations itself, and the
exported surface may still change as they land.
go/pkg/basecamp/eventfeed/loop.go:638
- Cancellation is not re-checked after
Load. A custom store that honors the supplied context can returncontext.Canceledwhen the caller cancels or callsClose, and this path then yieldscheckpoint_loadeven though cancellation is documented to end iteration cleanly. Match the mint/poll paths by givingrunCtxcancellation precedence over the load error.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93a3d5a457
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
93a3d5a to
308df85
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (5)
go/pkg/basecamp/eventfeed/scenario_fixture_test.go:737
Decoder.Moreis not a top-level EOF check: trailing]or}makes it return false, so malformed fixture input is accepted despite this helper's strict-decoding contract. Check the unconsumed bytes (or require a second decode to returnio.EOF) instead.
if dec.More() {
return fmt.Errorf("trailing JSON content")
go/pkg/basecamp/eventfeed/connector.go:535
- The restriction is broader than consumer callbacks.
TicketMinterandCheckpointStore.Load/Saveare also invoked synchronously on the run goroutine, so callingWaitfrom one of those implementations waits for itself; consequently “safe from anywhere ELSE” is incorrect and can lead custom seam authors into a deadlock.
// It is not callable from a consumer callback. Every callback — an observer, a
// signal handler, the loop body — runs ON the run goroutine, so waiting for
// that goroutine from inside one waits for itself. Close is the call that is
// safe from anywhere; this is the one that is safe from anywhere ELSE.
SPEC.md:2518
- This constraint must cover all code running synchronously on the connector's execution context, not only consumer callbacks. For example, a custom checkpoint store's
Saveruns there too and self-deadlocks if it callswait(); the cross-language contract should state the broader restriction.
`wait()` is not callable from a consumer callback, for the reason `close()` does not wait.
go/pkg/basecamp/eventfeed/scenario_conformance_test.go:15
- This header contradicts both the implementation and the new README disclosure: outbound frames (and saves outside the delivery witness) are queued and can satisfy a later expect step, so they are not arrival-strict. Please describe the tracked gap accurately here rather than claiming the driver follows the rule exactly.
// Strictness follows the family README's per-action-class rules exactly.
// Checkpoint saves and outbound frames are ARRIVAL-STRICT: a save carries the
go/pkg/basecamp/eventfeed/loop.go:1528
- The PR description still says a
durableGateprevents saves from commencing afterClose, but this implementation deliberately removed that gate, permits an accepted save afterClose, and usesWaitas the quiescence boundary. Please update the description (and the #784 status it cites) so reviewers and consumers are evaluating the lifecycle contract this diff actually implements.
// A position that reached here was ACCEPTED, and its events were delivered
// to the consumer before it was. It is written unconditionally, including
// after Close: the store is what tells the next run where those deliveries
// stopped, and dropping the write because the consumer closed a moment
// earlier silently re-delivers them. Ordering a second connector against
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2cb78a250
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…2 driver
The other half of the connector, stacked on the foundations PR. Everything here
is the running machine and its harness:
connector.go New, the options, Events, Close — the consumer surface
loop.go the state machine: states, transitions, timers, live buffer
catchup.go the poll walk, its page boundary, and the drain
recovery.go the 400/409/410 recovery matrix
scenario_* the tier-2 conformance driver, its fixture model, harness,
and self-tests
export_test.go the bridge from the external test package to the internal
observation points
Rebasing #705's 29 commits onto the foundations PR was attempted and abandoned:
8 of them touch ONLY foundation files and would replay empty, and 21 of the
remaining 22 mix both halves, so every one conflicts against the four fixes the
foundations PR applies (the credential boundary, the dedupe contract, the file
store, the redactor). Resolving 22 interleaved conflicts by hand is a worse
guarantee than re-cutting the tree, which is byte-exact by construction: the
foundation paths come from that PR's tip and these 17 from #705's.
The development history is not lost. It is tagged `pre-split/705-head` at
1646f2c1f, which is what the SHAs cited in #705's review threads still resolve
against, and the per-file reasoning it carries is unchanged in the code and its
comments.
Daybreak blocker 2. The walk took page.Position on trust, and an empty one
silently skipped history in two different ways depending on the entry class.
Position-resume: acceptPosition("") sets l.position = "", and entryCursor
selects on `l.position != ""` — so it does not preserve the old cursor, it falls
through to the StartResume default, which is a BARE PRESENT ENTRY. The feed
resumes at the server's head with everything between the real position and now
skipped, and reports nothing.
Present-class is worse. `held` uses "" as its sentinel for "the final entry was
not present-class", so an empty position is not saved as empty — it collapses
into the sentinel, `held != ""` skips acceptPosition AND saveCheckpoint
outright, and caught_up is announced anyway. The entry position is DISCARDED,
with the drain's deliveries already handed to the consumer and nothing durable
recording them.
Both are the same silent skip that a found-but-empty STORED position is already
refused for at load (d379f2e), arriving by the other door. The guard is the
same shape and in the same spirit, and it also makes saveCheckpoint's stated
contract true: it documents that "the live cursor is neither regressed nor
blanked", which the assignment one line above its call site did.
Terminal(poll_failed) is what §23 already classifies "an unexpected shape" from
the poll lane as, and it is not retried — a server that answered without a
position answers the same way to the same request.
The refusal precedes delivery, the counter resets, and every mutation of the
walk's state. That placement is the part worth pinning, so it is what the
mutation check exercises: moving the guard to AFTER the delivery loop fails
three of the four subtests on `delivered ids = [101], want []`, while the
"is not retried" subtest still passes — the subtests discriminate different
properties rather than all keying on one.
Removing the guard entirely fails all four, but by watchdog timeout rather than
on an assertion, and that is worth stating plainly: the un-fixed feed does not
misbehave visibly, it proceeds into Streaming as though nothing happened. That
IS the defect.
…s only Two independent items, both about what leaves the connector and when. #763 — Observer.Connected fired between the socket opening and the window that measures silence on it. Observer callbacks are host code on the consumer's goroutine (a log write, a metrics emission, an error-tracker breadcrumb), and newStaleHolder reads the window's origin at construction — so whatever the callback spent was time the window never counted, and a peer that went silent immediately got that much extra grace. §23 says the window arms at socket open; now it does. Starting the pump ahead of the callback is the same trade and harmless: it only fills a bounded channel this goroutine drains later. The test observes the ordering from INSIDE the callback, which is the only place it is visible — asserting on the timer set afterwards passes either way, because the set is identical and only its origin moves. Restoring the old order fails it with `timers armed when Connected fired = [handshake-deadline]`. Blocker 7 — the redactor from the foundations PR is applied to both URL-bearing observer surfaces. Observer.Gap carried the server's resume URL whole, and Cursor.PageURL carries the same URL at a walk announcement: an accepted 410 latches it as reconnect state (deliberately, so a socket torn down before the resume page cannot re-select the cursor the server already refused), so the reconnect announces its walk carrying it. Redacting Gap alone would have left the identical URL leaving through a different callback one reconnect later. The handler still receives the URL WHOLE and the resume poll still follows it whole, and both tests assert that pairing — a test checking only the observer would be satisfied by redacting both, which would break the disposition. Reaching the CatchUpStarted leak took the one sequence that keeps the latch alive. Dropping the resume poll's socket does not: transition 21 observes that at the page boundary, AFTER the resume page is accepted and saved, which releases the latch and reconnects at pos-1. Written that way first, it proved nothing; PollUnauthorized is what rides the reconnect cycle with the latch intact. The first version also raced the second walk's announcement, which -race reported. Observer.Disconnected(err) was audited and deliberately NOT redacted. An error is opaque text, and stripping a credential out of arbitrary text means modelling the credential — the one thing §23's "opaque bearer" contract forbids, and the trap dialFailure's own comment documents three review rounds of. Every error the connector puts there is its own sentinel or a seam error, and the built-in transport draws its causes from a closed vocabulary. The audit did find a real gap, and it is a contract gap rather than a code one: DialError.Err is supplied by the CableTransport and concatenated into Error(), and CableTransport is a documented extension point whose seam stated no obligation about the URL. So the obligation is now stated where an implementer reads it — on Dial, ReadFrame and WriteFrame — and Observer.Disconnected's doc says plainly that err is unredacted and why. Stating it on the seam is the only place it can hold; a guard here could not see inside an opaque error.
…e promise Daybreak blocker 6. Close's doc claimed "no seam call (mint, dial, poll) and no delivery can BEGIN after Close has returned". That is not true and cannot be made true: the run goroutine checks its context at each dispatch point and then acts, so a Close landing between a check and the call it guards cannot stop the call from starting — only from starting on a live context. Closing that window would mean holding a lock across every seam call, i.e. across arbitrary host code, trading a benign race for a deadlock reachable from any callback. So the promise is narrowed to what holds: cancellation is visible before Close returns, and anything that begins afterwards begins on an already-cancelled context. For reads that is self-limiting — the call returns promptly, its result is discarded, the run takes the Closed edge. One effect is not self-limiting, and it is the one now ORDERED. A checkpoint save is the connector's only effect that outlives the process, and cancellation cannot stop it: a CheckpointStore is entitled to ignore ctx, and the built-in FileCheckpointStore documents that it does, deliberately, so a position already accepted is not dropped by a shutdown race. The cancelled context Close publishes is precisely the signal that store is specified not to act on. durableGate closes it. A save holds the gate across the store call; Close latches the gate, waiting for at most one in-flight write. Either a save commenced before Close (and Close waits for it, which is right — its position was accepted and delivered before Close, and abandoning a durable write half-done is worse) or it does not commence at all. The failure that motivates it is sequential, not adversarial: Close, then a second connector over the same store, then a save the first had not yet begun landing on a lineage the new run has already loaded. It is a second lock, not mu, so a host's write cannot hold up a concurrent Close caller's cheap serialization; and it is released BEFORE either observer callback, because Close from inside a callback is documented as supported and holding the gate across host code would deadlock Observer.Checkpoint against the very save that fired it. That hazard is real, not theoretical: holding the gate across the callbacks deadlocks the Checkpoint subtest, and Go's timeout names it. Third item: a checkpoint LOAD that failed because the run was cancelled is no longer Terminal(checkpoint_load). The load happens on the first iteration and before the first mint, which is exactly the window a prompt Close lands in, and diagnosing the consumer's store for the consumer's own shutdown is wrong — §23 ends a closed iterator with no error element. Classified on the CONTEXT, not the error's shape, because a store is under no obligation to wrap ctx.Err(). The first version of that test closed BEFORE the run started, which never reaches the store at all — Events takes the isClosed latch with zero wire attempts — so it passed with the classification deleted. It now parks the load in flight and lands Close on it, and the paired test keeps an UNCANCELLED load failure terminal, so the guard cannot be widened into swallowing the real edge.
…tal scan #760, and the answer turned out to be removing a mechanism rather than adding one. drainScan returned the moment it found the deferral slot occupied, on the reasoning that "everything the pump has queued arrived behind this one, so the scan stops here either way". That is a claim about arrival ORDER, where §23's carve-out is a claim about which VERDICT governs: a raw invalid_event_stream_command observed during Draining is Terminal(protocol_fatal) with the drain not completed, the held position not saved and no caught_up. With any non-fatal outcome parked ahead of it the scan looked at nothing, the drain completed, the position saved and caught_up announced — all three of the things the carve-out exists to prevent. The budget that scan runs under is pumpDepth+1, sized in drain's own comment for the express purpose of reaching every frame the pump had already read. An occupied slot spent NONE of it. rev 15 selected a bounded deferral queue carved out of pumpDepth for this, and that is not needed. Reading the dispatch path, only ONE deferral is ever dispatched: dispatchDeferred sends it to pumpExited, dispatchDisconnect, or the invalid-frame teardown, and every one ends the cycle. A second entry could never be reached, so a queue would be a buffer sized for a delivery that cannot happen. The scan keeps the FIRST outcome — which is what preserves arrival order for the one that is reported — and discards the rest as it passes them. That removes the whole capacity question with it. Nothing new is retained, so the published memory bound ((pump depth + liveBufferCapacity) × MAX_FRAME_BYTES) is untouched, pumpDepth is untouched, and so is the depth at which the pump blocks — no fixture sweep, no §23 restatement, no ExportPumpDepth change. The new subtest needs both halves of the two around it and neither alone reaches the defect: deferring during the entry poll is what occupies the slot, and queuing the fatal frame MID-DRAIN is what puts it somewhere only the scan can find. Queued any earlier the ownership cut consumes it first and the carve-out fires without the scan being involved — which is how the first draft passed against un-fixed code. #758/#759, the bounded wait: no code change, and the reason is documented in place. The overshoot is real — a suspended firing re-arms a full window, so a deferral landing just after a frame waits close to two windows — but it cannot exceed two, because the post-select predicate runs on every wake and the first re-armed firing is necessarily past a deadline set one window after the deferral. A shorter re-armed window would be a staleness window that is not one, and a dedicated timer is the seventh kind §23 pins against, so the bound is stated rather than patched. I could not reproduce rev 15's missing-wake-source hang. Every path leaves one: a suspended firing always re-arms, and a latched authoritative expiry is reported by evaluate before any wait can consume its timer. The overshoot is what is real, and it is now published as the two-window worst case — carried to bc3 as an open §23 contract question rather than asserted as settled here.
Review of 4cff076 (B1 excluded, since uncommitted at the time). All four P1s reproduce; each fix is red-proven against the reported shape. P1 — Observer.Disconnected still leaked ticket text. Both arguments carry peer-controlled strings: a raw disconnect frame's reason, and a WebSocket close reason rendered through the error. Both were BOUNDED by §9's cap, which limits how much of a credential escapes rather than whether any does — the identical trap dialFailure documents three review rounds of. The cable server is exactly the party that knows the ticket: it was dialed with it. Both now go through closed vocabularies. observableDisconnectReason keeps the two reasons that change behavior and reports everything else as "other"; observableSocketError passes the connector's own sentinels and typed errors and degrades anything from a seam to a generic cause. CloseError.Error() renders only the code — an integer cannot carry a credential, and RFC 6455 codes are what an operator classifies on; Reason stays a readable FIELD. A canary planting a ticket in every peer-controlled teardown string found MORE than was reported: raw seam read errors leak too, which seam documentation cannot repair because the connector forwarded them verbatim. Four arms, all red before and green after. P1 — durableGate deadlocked reentrantly and blocked Close. It held the lock across CheckpointStore.Save while Close waited for it: a store whose Save calls Close self-deadlocks on the caller's own goroutine, and a merely stalled store blocked EVERY Close indefinitely — contradicting the one thing Close promises unconditionally. The two promises could not coexist, so the waiting one is dropped: the gate is claimed and released atomically, Close latches and returns, and a save that already claimed still completes. The guarantee is unchanged in substance — no save COMMENCES after Close returns — with commencing defined as claiming the gate, which takes no host code with it. The old test asserted Close WAITS and is replaced by one asserting it does not; the gate-holding variant deadlocks the new test at 40s. P1 — Close precedence, reopened by #763. Arming staleness before Connected also starts the pump before it, so a fatal frame can already be queued when a Connected callback calls Close, leaving two ready select cases. Reproduced: 25/50 rounds emitted a terminal element after Close returned. Fixed at the ONE exit (emitTerminal) rather than per-select — many selects, one exit, and a rule every future select must remember is what produced this. P1 — B2 discarded an earlier socket verdict. A deferred protocol-fatal followed by a positionless page took poll_failed, because disposal clears the deferral. The failed-poll branch already dispatches the deferral first, with a comment giving this exact reason; the new guard did not follow it. Now it does. Also fixed: TestNoCheckpointSaveCommencesAfterClose was vacuous (it closed before the run reached a page) and now closes from Observer.PageDelivered, the callback immediately preceding the save; the cancellation check after the checkpoint load covers every result rather than only the failure, since a found-empty result became terminal and a successful one let the run fire Connecting after Close; and deliver()'s stale "no delivery begins after Close returns" claim is corrected in place — it is a check-then-act, and the honest guarantee is the one Close states. Two of these touch foundations files that belong to #777 — CloseError in seams.go and its test. They stay here because the leak is only observable through the loop's observer path, which is this PR's, and the canary that proves it lives here. TestCloseError_Message is INVERTED, not adjusted: it required Error() to render the peer's reason, so it pinned the wrong contract. Verified: build, vet, -race, 22/22 fixtures, go-lint 0 issues, gosec 0 issues.
All three are documentation-accuracy fixes with no behaviour change; the verification is that each comment now matches the code beside it. Observer.Disconnected still promised that err is "passed through UNREDACTED". That stopped being true when the ticket-leak fix routed both arguments through closed vocabularies: observableDisconnectReason collapses every unrecognized peer reason to "other", and observableSocketError replaces every unrecognized error with errSocketFailed. The old text told a custom-transport author their diagnostics reach the observer, which is exactly backwards -- so the corollary is now stated outright rather than left to be discovered by an author whose error vanished. go/README.md claimed the connector "performs no wire I/O of its own" two sentences after describing the Action Cable subscription it dials. AGENTS.md and doc.go both name that dial as the one sanctioned non-HTTP wire act; doc.go's version of the sentence already carried the qualification and the README's did not. Scoped to HTTP API I/O, with the dial named. durableGate's opening paragraph promises to prevent a replacement connector overwriting a lineage, and "begun" there is the gate's term of art for claiming the gate -- defined twenty lines below the sentence that leans on it. A reviewer read it the other way, which is the evidence it needed saying. The residual window is real and narrowed rather than closed: claim and store call are not atomic, so a descheduled save can still land behind a replacement and move a checkpoint backward. Closing it needs Close to wait (a deadlock from any callback) or a fencing token on CheckpointStore (a six-SDK contract change), so it is stated here and tracked in #784.
observableSocketError exists so the connector does not DEPEND on a custom
transport honoring the seam's no-cable-URL obligation. For four of its arms it
did depend on exactly that.
Matching a sentinel is not the same as being one. errors.Is walks the chain and
reports; it does not extract. So a transport returning
fmt.Errorf("read %s: %w", cableURL, context.Canceled)
matched the recognized arm, and returning the argument handed the wrapper --
ticket and all -- to Observer.Disconnected, which is a logging surface. The
credential rides in the URL the peer was dialed with, so the one place text is
most likely to be annotated is the one place the vocabulary was open.
The typed arms below were already safe by a different mechanism: errors.As
assigns the inner value, so returning it drops the wrapper. That asymmetry is
why the two halves look different and must, and the test pins the As half too --
a refactor unifying both onto errors.Is would silently reopen this.
Each arm now returns the canonical sentinel rather than its argument. errors.Is
still matches for a consumer that branches on it; what does not survive is the
wrapper's text.
Red-proven with only the switch reverted: all four sentinel cases FAIL with the
literal ticket in the message, while the wrapped-CloseError case PASSES against
un-fixed code -- so the test is not vacuously red, and it discriminates the two
mechanisms rather than asserting a blanket property.
Also corrected, wrong in the same direction: Observer.Disconnected's godoc had
just been rewritten to claim the vocabulary needs nothing of a custom transport,
and the PR description still said the callback was "audited and deliberately not
redacted". Both now describe what the code does.
CableTransport.Dial told implementers its returned error "reaches Observer.Disconnected, which hosts log". It does not, and cannot: a dial failure has no socket, so there is no teardown to report. DialPolicy becomes Terminal(invalid_cable_url); every other kind takes transition 7 to backoff, where the connector reports the classification and drops the cause. Left alone, that reads as a licence to relax -- the callback never fires, so why redact? The real exposure runs the other way and is stronger. A DialPolicy error is yielded as the ITERATION's terminal error: the consumer's own error value, which they cannot decline to receive, where a callback is something they may never have registered. So the no-credential obligation is not softened by the correction, it is what keeps a ticket out of the error a caller gets back from Events. ReadFrame's identically worded claim is left as it stands, because for a read error it is true. Fourth of these today -- prose that was accurate when written and falsified by a later, correct change. Noted on the PR as a pattern rather than filed: this package states unusually precise contracts in comments and nothing checks them, so each correct fix quietly ages the paragraph above it.
… enforces
SPEC §23 gives the feed exactly one final error element, and that element ends
iteration. The driver checked neither half. It recorded a terminal and kept
ranging, overwriting `terminal` each time, so three non-conforming shapes
passed every terminal fixture identically to a compliant connector:
- yielding the expected terminal reason TWICE (the second overwrote the
first with an equal value, so every later assertion still passed);
- delivering an event AFTER the error element;
- never ending iteration at all -- the deferred cancel ended the range
afterwards, so nothing downstream could tell an iteration that ended by
itself from one that had to be cancelled.
None of the three is observable from an end-state assertion, which is why the
gap survived: the fixtures in the tree pass either way. A test that cannot
fail the contract it claims to enforce is worse than no test, because it is
counted as coverage.
The count and the post-terminal delivery are recorded as violations, which
block every subsequent await, so a scenario fails at its next step rather than
silently at the end. The first element is retained rather than the latest --
"exactly one final element" means the first one is the real one. Requiring the
iteration to have ended is a driver-side await, bounded by the same watchdog
as every other rendezvous, so a connector that stays open fails instead of
hanging.
Red-proven per case: with the three guards reverted, all three hostile shapes
report "the driver accepted ..." and the compliant shape still passes, so the
probes discriminate rather than failing on everything. All three were
reachable; none had to be dropped as unreachable.
Found by Copilot as a suppressed comment, which is the second time today that
block carried the sharper finding.
…'s to write
`observableSocketError` exists so that `Observer.Disconnected` — a logging
surface whose destination the connector knows nothing about — carries only
values the connector authored. Recognition is by type, which is only as strong
as the type's authorship, and two arms failed that test.
`*DialError` and `*TerminalError` are both exported with exported fields, and
both render free text: DialError its `Reason` plus `Err.Error()`, TerminalError
its `Msg` plus `Err.Error()`. `CableTransport` is a documented extension point,
so a `CableConn.ReadFrame` returning `&TerminalError{Msg: cableURL}` reached the
callback verbatim — the ticket rides in that URL's query string, and §23 calls
the ticket an opaque bearer credential that is never logged. Neither passes
through now.
The three arms that remain are safe by construction rather than by convention:
the sentinels are package-level values whose text is fixed in this file,
`*CloseError` renders an integer code alone, and `*invalidFrameError` renders
one of two shape constants. Nothing is lost on the live path — dial failures
never reach this function, and a terminated feed's reason and detail arrive
through the iteration's terminal element, which is the semantic channel and
keeps its SPEC-mandated wording. `seams.go`'s contract said the removed types
were preserved; it now states the closed vocabulary and points at the terminal
element for the rest.
The canary arm that should have caught this was a false negative three times
over, and each fault alone was enough. `{"type":"message",...}` is not a
correlated broadcast — that shape is the TYPELESS one — so it never reached
`decodeMessageEvent`. The payload was truncated JSON, so it failed the envelope
unmarshal first and the payload was never read. And the canary sat in
`identifier`, which nothing renders. It now serves a well-formed typeless
broadcast under the connector's own identifier with the canary in `created_at`,
the one field whose decoder quotes its input back.
Four arms added for the errors a seam can author, at both seams that produce
them: ReadFrame for the two types, and WriteFrame for the subscribe write —
which had to be armed before welcome, since that is when the write happens.
The assertions gained an identity check, because "the canary is absent" is
satisfied by a callback firing with a nil error, which is not the guarantee.
Mutation confirms the whole set: restoring the two arms fails nine cases, and
three of them fail ONLY on the identity assertion.
#760 established that a protocol-fatal disconnect the pump has ALREADY READ must outrank whatever non-fatal outcome happens to be parked ahead of it, and fixed that inside the drain. The obligation is not the drain's. SPEC §23 says so twice — "terminal from EVERY socket-open state", and "a raw invalid_event_stream_command frame READ in AwaitingWelcome, CatchingUp, or Draining ... never Backoff. An explicitly non-retryable protocol rejection must not reconnect from any state." READ is the boundary the obligation attaches to. The walk has four points where it disposes of a deferred receive, and all four dispatched with the pump's queue never inspected. A `remote` disconnect in the slot — recoverable, arrived first — beat a fatal sitting right behind it: the cycle ended as an ordinary socket failure and the connector re-minted against a server that had already refused the command. Confirmed at every boundary before the fix, by mint count rather than by terminal reason alone, since a connector that reconnects once and then terminates on the re-issued frame still ends at protocol_fatal. `drainScan` becomes `fatalScan`, called at all four boundaries through `probeFatal` with a fresh pumpDepth+1 budget. That bound is the drain's, and for the drain's reason: the queue holds at most pumpDepth, the pump may be blocked in hand-off with one more, and the ownership cut's liveBufferCapacity is caller-configurable down to 1 — one queued ping under WithLiveBufferCapacity(1) spends an entire admission pass, which is what the page-boundary case reproduces. What the scan does with what it dequeues is what makes it safe mid-walk: live events are admitted to the buffer, which is what CatchingUp does with them anyway; the FIRST non-fatal socket outcome goes to the same single slot the caller is about to dispatch from, preserving arrival order for the outcome actually reported; and later ones are discarded, which is all that could happen to them, since every dispatch of a socket outcome ends the cycle. One case per boundary, and the mutation matrix is a clean diagonal: removing the probe at any one site fails exactly its own case and no other. Two of the four needed the case built for them — the page boundary only exposes its probe under a buffer capacity of 1 with a frame wedged ahead of the fatal, and the superseded boundary needs the call abandoned rather than returned. TestSupersededPollBoundSurvivesPumpBackPressure needed the harness's frame-handled drain that the blocked-hand-off test already carries: the notification send is blocking and a probe handles a whole queue in one pass, so the loop wedged in the test's own callback. That inline workaround is now a harness helper, since any test that fills the queue needs it.
… guarantee
`durableGate` is deleted, and `Connector.Wait` replaces what it was reaching
for.
The gate refused a checkpoint save once Close had latched. Its own doc states
the residual honestly: claiming the gate and writing are not atomic together,
so the window in which a replacement connector's newer position gets
overwritten is NARROWED, from [decision, write] to [claim, write], not closed.
Weigh that against what it cost, which the test pinning it spelled out without
naming it as a cost. Close taken from Observer.PageDelivered — a callback that
fires immediately after the page's events reach the consumer and immediately
before its position is written — meant that position was never persisted. The
next run re-delivers those events with no record they ever arrived.
So: a bounded-replay harm avoided some of the time, paid for with a
bounded-replay harm caused every time. And it prevented at most about one save
anyway, since runCtx is cancelled synchronously and the walk's dispatch points
unwind almost immediately.
The guarantee it wanted is free. The iterator returning IS quiescence — the run
has exited, so no save can be in flight, by construction rather than by a window
narrow enough to be unlikely. `Wait` exposes that to a caller who closes from a
goroutine that does not own the range loop; a caller that owns the loop already
has it. The contract is now "await termination — or Wait — before opening a
second connector over the same store", which drops the term of art ("commence")
that two reviewers independently read as a stronger promise than it was.
Closes #784.
The second half is Close precedence, which `emitTerminal` had established for
the terminal element and nothing else. Every lifecycle announcement kept firing
after a Close, because the nearest runCtx check sat AFTER the callback — placed
to catch a Close taken from INSIDE it, which is a different Close from one taken
earlier. Five checks now, each with a case that only it can satisfy: before
`connecting`, `catch_up_started` on both the post-confirmation and repair paths,
`caught_up`, the ownership cut, and socketCheck.
Two of those are about announcements the walk makes on the way past. The
ownership cut and socketCheck both DISPATCH what they dequeue, so a disconnect
frame queued when the consumer closes was announced through
Observer.Disconnected — a teardown report for a socket the consumer had already
stopped caring about.
Ordering within the page boundary is deliberate and runs the other way from the
rest: the page's save happens BEFORE the check, its announcements after. A check
placed before the save would reintroduce exactly the dropped save the gate was
deleted for. Finish the page, then observe the close.
Getting the mutation matrix to a clean diagonal took three corrections worth
recording. The `caught_up` check and the page boundary's masked each other under
a position-resume entry, so the case that isolates `caught_up` had to use a
PRESENT-class entry, where the held save fires after the walk has returned. The
socketCheck case was driving the ownership cut instead — the default harness
resolves to present-class — and claimed in its name to be about the other. And
its hand-off wait was satisfied by a stale token from the handshake, so Close
landed before the frame was ever queued and the code under test had nothing to
find.
One check has no test and says so: the repair path's, which backstops a select
in `stream` where `runCtx.Done()` and `repair.C()` can both be ready and Go
chooses. That is the same race `emitTerminal` documents, and it cannot be driven
deterministically from outside.
…ontract `wait()` is a public act with a cross-SDK obligation, not a Go convenience, so §23's Consumer Surface states it rather than leaving the other five SDKs to infer it from the Go source. What the section said about `close()` was true and incomplete in the same breath: it abandons, it is callable from anywhere — and it silently implied that closing was enough to hand a checkpoint store to a replacement connector. It is not, and no gate inside `close()` can make it so. So the section now says both halves: a save decided just before the close may be written just after, deliberately, because its events were already delivered; and `wait()` is the quiescence point, with the iteration terminating being the same guarantee for a consumer that owns the loop. The naming note says what has to exist rather than what has to be added. A language whose streaming idiom already exposes the run's completion — a Kotlin Job, a Swift Task — has `wait()` already; the requirement is that observing the run's exit be possible WITHOUT `close()`, since `close()` is the one call that cannot wait.
…es not have Copilot's two suppressed comments on this PR are right. The README states, normatively and for all six drivers, that checkpoint saves and outbound frames are ARRIVAL-strict — "observed while the current step is anything other than their matching expect step, the scenario fails immediately." The Go reference driver appends both to queues and lets the matching expect step consume whatever is there, with no record of which step was active on arrival and no field to record it against. That is the order-strict rule, which is what the parked classes get. The rule is not weakened to match the implementation, because the rule is normative for five drivers not yet written and this is one implementation behind it. What the README now carries is the gap itself, sized: early-SAVE ordering is not at risk — every save records the delivery count it arrived at, and fixture 12 fails a save that precedes its page's deliveries — so the residue is an outbound frame written earlier than its step, concretely a subscribe command sent at dial rather than on welcome. Not patched here because the fix is not the queue check it looks like. The rule's own second sentence is the hard part: an action that becomes legal the instant a rendezvous is satisfied must match the NEXT step rather than fail against the stale one, so the handoff between adjacent steps has to be atomic against the harness mutex. A naive active-step guard fails exactly there and would turn every rendezvous-adjacent action into a false failure. Filed as #792 with the red proof it needs: a connector that subscribes at dial passes all 23 fixtures today.
Four defects from daybreak's review, each one a place where an argument I made correctly in one spot was not carried to the next. **A seam-authored DialPolicy verdict reached the terminal element.** The previous commit established that *DialError and *TerminalError are exported, seam-constructible, and render free text, and used that to strip them from Observer.Disconnected — then exempted the terminal channel on the grounds that "terminal elements keep their SPEC-mandated text". That is true of the one case §23 actually mandates, filter_invalid, and false as a blanket. A DialPolicy error comes out of CableTransport.Dial, a documented extension point, and its Reason and cause went verbatim into the terminal's Msg and chain. The KIND is still read; the text is now the connector's own and no cause is retained. The connector's own pre-check keeps its text and is not the same case: its reasons are written in transport.go, and its two interpolations are a URL scheme and a port, which url.Parse constrains to alphanumerics and digits. **CloseError passed through with its peer Reason intact.** The comment said "renders only its code — an integer cannot carry a credential", which is true of Error() and irrelevant to the struct: Reason is exported and held the peer's string, so %+v, a structured logger, or ce.Reason put it on the surface the closed vocabulary exists to protect. Rendering is not the only way out of a struct. It is rebuilt code-only now. **The accepted-page save was handed a cancelled context.** Deleting the durable gate was supposed to make an accepted position durable even when Close lands at the save boundary. It passed l.runCtx, which Close cancels synchronously, so a store that HONORS its context — which CheckpointStore permits and says nothing against — returns ctx.Err() and the position is lost, exactly as under the gate. Both stores this package ships ignore ctx, which is why every test passed: the guarantee was tested only against implementations that could not fail it. context.WithoutCancel keeps the run's values and drops the cancellation. The trade is named in the code: a store that blocks forever now blocks the run's exit rather than being released by Close, which is the lesser harm, and the narrower one, since a store may already ignore ctx entirely. **The fatal scan could sample inside the pump's read window.** The carve-out is stated over "every frame the pump had ALREADY READ", and pumpDepth+1 delivers that only for a frame blocked inside the hand-off with a full queue. Between ReadFrame returning and the send starting, the frame is invisible: the scan sees an empty queue, completes the drain, saves the held position and announces caught_up with the fatal sitting in the pump's stack. Red-proven with a new pumpRead hook that parks the pump in exactly that window — checkpoint ledger [pos-1] where the carve-out requires none. Peeking at what the pump holds would move the race one instruction earlier rather than close it, so the scan's completion condition changed instead: an empty queue ends the scan only when the pump also holds nothing it has read. That second half is an atomic flag covering a stretch with no I/O and no blocking except the hand-off, which the scan's own dequeue releases, so the spin terminates and the budget bounds it regardless. Mutation: all four die, each on its own test and no other. The fourth needed its mutant rewritten — removing the guard alone left `runtime` unused, and a vet failure is not a kill.
§23 said a save decided just before close "may still be written just after" because "a store is entitled to ignore the cancelled context" — which describes the accident that made the Go implementation work rather than the rule the other five SDKs need. A store that HONORS its context is equally compliant, and against one the guarantee evaporates: the position for events already delivered is dropped, silently, and the next run re-delivers them. So the rule is now the detached context, stated with its trade: values carried, cancellation dropped, and a store that blocks indefinitely delays the run's exit — and `wait()` — instead of being released by `close()`. Bounded by the store's own behavior, against re-delivery that is bounded by nothing.
…ary moves The previous commit claimed the fatal scan covered "every frame the pump had ALREADY READ" and added an atomic flag to deliver it. daybreak is right that it does not: the flag is published AFTER the read returns, leaving a pre-publish window, and the scan loads it AFTER its own empty select, leaving a second window in which the frame lands and the flag clears between the two reads. Two samples are not a happens-before, and the hook I added to test it sat after the store, so it excluded both interleavings it was supposed to catch. Closing the window for real needs the read to complete inside a critical section the scan can enter. The read blocks indefinitely on a quiet socket, so that lock deadlocks the drain against a peer that stopped talking — which is why this is a narrowed boundary rather than a third sample. So the flag, its hook, and its test are gone, and §23 now states what every implementation must actually cover: everything HANDED to the state machine, plus the one frame a blocked hand-off is holding. "Observed" was always the spec's word; the reader is a single goroutine, so the plus-one is exactly one, and it is why the budget is pump depth + 1. The test drives the guarantee through the hand-off rendezvous rather than through a window no scan can see. Two more from the same review, both current-head: A dial returning (conn, err) leaked the connection — legal in Go, and easy for a transport that fails mid-handshake — which under the reconnect cycle is a leak per backoff round. It is closed before the error is classified. A (nil, nil) return now takes the transient failure edge instead of installing a nil CableConn and panicking on the first read. The detached save's test could not tell WithoutCancel from Background: both survive cancellation, and only the first keeps the caller's context values — a trace span, a tenant, a request id. The store now records what it saw and the test asserts it, which kills the Background mutant for that reason alone. doc.go still said FOUNDATIONS ONLY with no Connector and no run loop. It names the one piece genuinely outstanding instead: the Layer-1 adapters, and the two obligations that ride on them rather than on anything here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fe0a8798a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rival-strict `conformance/event-feed/README.md` binds every driver to one rule: a checkpoint save or an outbound frame observed while the current step is anything other than its matching expect step fails the scenario there and then. The Go reference driver did not do that. `Save` appended to a queue, `readPeer` appended to another, and the matching expect step later consumed whatever was sitting there — so an action emitted during an earlier, non-matching step still satisfied its expectation. There was no active-step field to judge against, and the README carried a paragraph saying so, which this deletes. The rule's own second sentence names why this is not a queue check. The connector reacts to a rendezvous being satisfied before the driver has stepped over it, so a pointer compared naively reports every rendezvous-adjacent action as an early arrival — not an edge case: with the read-through below removed, five of the twenty-two fixtures fail outright. Three things make the handoff atomic with respect to the harness mutex. `await` advances the pointer inside the critical section that observed the satisfying state, which covers a driver blocked ON the rendezvous. `currentStepLocked` reads through steps the recorded history has already satisfied but the driver has not consumed, which covers a driver that has not reached the rendezvous at all — a poll's page is delivered and then checkpointed by one causal chain in the connector, and the driver need not have run its `expectDelivered` step by the time the save lands. And a step the driver PERFORMS rather than waits for hands the pointer on before it acts, since nothing the connector does can precede the act; that is also what stops the read-through dead at an action step the driver has not performed, which is what keeps a subscribe written at dial — before the `serve welcome` that legalizes it — an early arrival rather than a match on the step after. The judgement is made before the action is recorded, so the matching step's own "one of mine is waiting" condition is still false and the read-through cannot run past it. `expectConnect` now takes the dial and installs the socket under the mutex, because the lookahead reads `d.peer` and `d.connectsTaken` from the connector's goroutines. Fixture 02's reordering probe now fails on arrival rather than on the save's delivery-count witness, which is the stronger diagnosis of the same defect. The witness stays anyway: the read-through is deliberately permissive — its job is only to say where the script has got to, and erring strict would stall the pointer and fail a compliant connector — so the witness is what still speaks if an arm is ever loosened. What the rule cannot reach in this lane is worth writing down, since five more drivers follow. A save is observed where the connector makes it, but an outbound frame is observed when the harness reads it off the loopback socket, a hop behind the write. A connector mutated to write its subscribe at dial is read only after the driver has served the welcome and stepped on, so no fixture catches it on arrival — 8 of 8 runs, with and without the action-step handoff. Fixture 07 catches that mutant structurally instead, as a frame where the script expects a close, and now names the step it belonged to. The rule is not the weaker half; the observation is, and pinning the write's own instant would need a witness at the write rather than a stricter reading of the rule. The driver's own tests pin each half against the mutation that deletes it, on the harness record paths where the rule lives, with every legal case paired against the illegal one it differs from by a single fact — including the subscribe-at-dial shape, which is deterministic there even though the socket makes it racy in a fixture.
…ait could not hear #758. Every socket-open wait in the connector carries the staleness cases; pollPage did not. A socket that goes silently half-open produces no frame and no read error, so the firing is the only evidence there will ever be — and it sat unconsumed while a PollSource that returns only on cancellation held CatchingUp forever, with nothing able to cancel it, because the cancel would have come from the teardown the wait was preventing. Both reviewers prescribed disposing the attempt where the expiry is observed, and that breaks a pinned ordering: TestWalkFailureBetweenPages/staleness_expiry advances a full window while the first poll is in flight and requires the walk to still accept, deliver and SAVE that page. It is transition 21's finish-the-page rule, and it is load-bearing — the reconnect re-enters at pos-1 rather than replaying. That mutant is in the record: checkpoint ledger = [], want [pos-1]. So the expiry is treated as what it is, a socket outcome, and takes the same route as a disconnect frame: deferred into the single slot, dispatched at the page boundary. The slot gains a third form, carrying the age rather than a frame, and the protocol-fatal probe excludes it by name rather than trusting what parsing a zero item happens to return. That left the bounded wait unbounded again, which is the part worth explaining. The wait relies on the staleness timer's firings and re-arms as WAKE-UPS. An authoritative expiry latches, and nothing re-arms over a latch — that is deliberate, and it is what keeps a frame arriving after the deadline from erasing the verdict — so consuming the firing here consumed the last wake the wait had. graceWake re-arms the same `staleness` kind purely as a wake source, keeping the latch so the pump's per-frame reset and a blocked hand-off's release cannot touch it. The outstanding set is untouched: {staleness} stays {staleness}, and §23's six kinds and per-state exact sets are not a thing a Go fix gets to change. The bound is now published rather than living in a Go comment. Worst case is DETECTION WINDOW + GRACE PHASE, both one staleness window, and the grace phase is a deadline read at the instant of deferral with two immunities that are the whole point of naming it: a frame arriving inside it must not extend it, and a blocked hand-off must not suspend it. The second is what the earlier borrowed bound failed — this wait is the one that deliberately stops draining, so the full-queue premise the suspension rule rests on is false here by construction. awaitSupersededPoll's "close to two windows ... cannot exceed two" paragraph is replaced by that statement rather than deleted; it was describing an accident of borrowing the timer, where this is a contract for six SDKs. Six tests, each able to fail alone. Four of them were written first and shown to hang against the unfixed code, reported as bounded watchdog failures rather than a wedged binary. The immunity pair needed more care than expected: a frame inside the grace phase hands the wait a WAKE as well as a re-armed window, and a wake alone is enough to carry an implementation that also moved its deadline to a correct-looking teardown — so the test consumes that wake first, through a new hook, and then requires the phase to finish on a wake of its own. Without that, the no-latch mutant survived. One guard is honestly unkillable and stays anyway. Dropping the probe's `!d.stale` changes no behavior today, because protocolFatalFrame rejects a zero item through parseFrame failing on nil bytes. That is an accident of an unrelated function's error handling, and a probe resting on it is one refactor away from reading a staleness expiry as a disconnect frame.
Four threads arrived on the push. One is stale — the connection returned alongside a dial error is already closed before classification, two commits back. The post-confirmation welcome is real and filed as #800, with the reason it is not a one-liner: that dispatch arm has no bounded-write discipline, and §23 does not say whether a retransmit re-arms the confirmation deadline. The nil dial result was mine, introduced when I added the (nil, nil) guard two rounds ago. Every other exit from that select stops the handshake deadline; this one returned without doing so, carrying an armed handshake-deadline into Backoff — whose exact outstanding set §23 pins at {backoff} — and a transport returning (nil, nil) repeatedly accumulates one ghost timer per cycle, each firing later into a state with no edge for it. The test asserts the exact SET rather than the absence of a panic, because a defensive edge that leaves a timer behind has only traded a crash for a leak. Mutation confirms it: map[backoff:1 handshake-deadline:1]. The terminal race is real and narrower than it reads. Close latches isClosed and THEN cancels, both inside one critical section, so a reader of runCtx landing between those two statements sees a live context while Close is already committed and about to return — and publishes an error element to a consumer that closed. emitTerminal now decides by claiming under the mutex that owns the latch, which has no such interval: the claim runs wholly before or wholly after Close's critical section. What it does NOT do is stop a terminal from being published after Close returns, and the shape it replaced did not either — a claim that wins publishes, and Close was then called against a feed that had already terminated. That is correct, and worth stating so the next reader does not read more into the change than it does. No test discriminates it, and the first one I wrote passed against the code it was meant to fail — the hook sat where both versions behave identically, which is the same placement error I criticised in the pump-read hook a round ago. It is deleted rather than kept as decoration. Reaching the real window needs a hook between two adjacent statements inside Close, existing for nothing else, and even then the behaviour it replaces is racy rather than reliably wrong. The change is kept because deciding against the authoritative latch is the correct shape, not because a proof forced it.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.
Suppressed comments (1)
go/pkg/basecamp/eventfeed/catchup.go:520
- A non-staleness outcome deferred late in the current staleness window does not receive a fresh grace-phase wake. When the pre-existing timer fires,
awaitSupersededPolltreatsevaluate(...) == okas the verdict and returns immediately, even if this fixeddeadlineis still almost a full window away. That contradicts the documented full grace phase measured from deferral and can abandon a poll prematurely. ArmgraceWakefor every deferred socket outcome (not only an already-fired staleness outcome), so the wake is fixed at this new deadline and cannot be inherited from the older detection window.
deadline := l.cfg.clock.Now().Add(l.cfg.staleAfter)
if stale {
// The firing this branch consumed was the wait's own wake source,
// and an authoritative expiry latches — so without this there is
// nothing left to wake the grace phase at all, and the bounded wait
// becomes unbounded again. Armed BEFORE the hook, so a test that
// rendezvouses on the deferral cannot advance past a wake that is
// not yet armed.
at.lc.stale.graceWake()
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.
Suppressed comments (3)
go/pkg/basecamp/eventfeed/connector.go:495
- The PR description still says
durableGateprevents a checkpoint save from commencing afterCloseand frames #784 as the remaining post-claim race. This implementation intentionally removes that guarantee: a pre-close decision may start writing afterClose, andWaitis now the ordering primitive. Please update the Blocker 6 / #784 narrative so reviewers and release notes describe the lifecycle contract actually shipped here.
// just before Close can still be written just after. That is intended. The
go/pkg/basecamp/eventfeed/scenario_conformance_test.go:502
- A schema-valid
expectPollmay contain bothurlandquery, but this early return silently skips all query/filter assertions wheneverurlis present. Such a fixture can therefore pass even if the connector supplies the wrong filters while following the correct continuation URL. Validate the URL and then continue into query validation when both pins are supplied.
if step.URL != "" {
if call.cursor.PageURL != step.URL {
return fmt.Errorf("the poll targeted %+v, want the URL followed verbatim: %s", call.cursor, step.URL)
}
return nil
go/README.md:688
- This lifecycle summary omits the new quiescence requirement.
Closecan return while a detached checkpoint save is still pending, so a consumer that immediately opens a replacement over the same store can race that save. Document that callers must await iteration termination or callWaitbefore replacement; otherwise “the feed is resumable” suggestsClosealone provides that ordering.
`Close` stops the feed without draining, and cancelling the context, calling `Close`, or
breaking out of the loop all end iteration with **no** error element — a clean stop, and
the feed is resumable by design.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 049cfc5ca2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // even then the behaviour it replaces is racy rather than reliably wrong. | ||
| // It is kept because deciding against the authoritative latch is simply | ||
| // the correct shape, not because a red proof forced it. | ||
| if !l.cfg.claimTerminal() { |
There was a problem hiding this comment.
Honor caller cancellation before publishing a terminal
When the caller cancels ctx after a terminal outcome is selected but before emitTerminal runs—for example, while a subscription-rejection teardown is waiting for the socket close—claimTerminal checks only the connector's isClosed latch, so this still yields the terminal error. Caller cancellation is documented as the same clean, no-error universal edge as Close; fold cancellation into the terminal-publication ordering rather than allowing an already-canceled run to emit an error.
Useful? React with 👍 / 👎.
| if err := checkOriginScheme(canonical); err != nil { | ||
| return err | ||
| } | ||
| if err := checkIdentityText("the API base origin", canonical); err != nil { |
There was a problem hiding this comment.
Validate origin UTF-8 before canonicalizing it
When an origin contains invalid UTF-8 in its hostname, url.Parse accepts the bytes and CanonicalOrigin passes them through strings.ToLower, which replaces each invalid byte with U+FFFD; validating only the resulting canonical string therefore succeeds. Distinct inputs such as hosts containing \xff and \xfe collapse to the same canonical origin and consequently the same checkpoint identity, allowing separate configured origins to load and overwrite one lineage; validate cfg.origin before this lossy canonicalization.
Useful? React with 👍 / 👎.
| // Transition 12: always terminal — cancel the deadline, explicitly | ||
| // close the still-open socket (Action Cable leaves a rejected | ||
| // socket open), ZERO reconnects. | ||
| l.disposeAttempt(at, *deadline) |
There was a problem hiding this comment.
Check the deadline before accepting a rejection
When a matching reject_subscription and the confirmation deadline are both ready, the frame can win the select and this teardown ignores the deadline timer's failed Stop, producing the permanent subscription_rejected terminal instead of transition 14's teardown and reconnect. Apply the same expired-timer ordering check used by the neighboring confirm_subscription branch so a rejection processed after its confirmation window cannot terminate the feed.
Useful? React with 👍 / 👎.
The Go reference implementation of the SPEC.md §23 Event Feed connector — BC3's
account-wide event feed over Action Cable push plus polling catch-up — and the
tier-2 conformance driver. All 22 fixtures pass (23 with #778's addition).
Layer 2 only: the connector reaches the wire through
TicketMinter/PollSourceseams and one sanctioned cable dial (AGENTS.md Hard Rule 2). TheLayer-1 adapters are deferred to G1b, so the package is experimental and a
consumer supplies the seams today.
What is here
connector.go(New, the options,Events,Close),loop.go(states,transitions, timers, live buffer),
catchup.go(the poll walk, its pageboundary, the drain),
recovery.go(the 400/409/410 matrix), and the tier-2driver with its fixture model, harness and self-tests.
The foundations — seams, wire types, transports, filters, checkpoint identity,
the file store, dedupe, backoff, clock, cable codec,
feedtest/— are #777.Rebase note
The 29 commits were re-cut, not rebased. Rebasing onto #777 was attempted
and abandoned: 8 of them touch only foundation files and would replay empty, and
21 of the remaining 22 mix both halves, so every one conflicts against the four
fixes #777 applies. Resolving 22 interleaved conflicts by hand is a worse
guarantee than re-cutting, which is byte-exact by construction — the foundation
paths come from #777's tip, these 17 from the pre-split head, verified with an
empty
git diffagainst both.The five fixes on top
Blocker 2 — a poll page carrying no position is malformed
The walk took
page.Positionon trust, and an empty one silently skippedhistory in two different ways.
Position-resume:
acceptPosition("")setsl.position = "", andentryCursorselects onl.position != ""— so it does not preserve the oldcursor, it falls through to a bare present entry. The feed resumes at the
server's head with everything between skipped, reporting nothing.
Present-class is worse.
helduses""as its sentinel for "the final entrywas not present-class", so an empty position is not saved as empty — it
collapses into the sentinel, the
held != ""guard skipsacceptPositionandsaveCheckpointoutright, andcaught_upannounces anyway. The position isdiscarded, with the drain's deliveries already handed to the consumer.
Refused before delivery, counter resets, and every mutation. That placement is
what the mutation check exercises: moving the guard after the delivery loop
fails three of four subtests on
delivered ids = [101], want [].Blocker 6 — order the one durable effect against Close, and narrow the promise
Close's doc claimed "no seam call and no delivery can begin after Close hasreturned". That is not true and cannot be made true: the run goroutine checks its
context at each dispatch point and then acts, so a Close landing in between
cannot stop the call from starting — only from starting on a live context.
Closing that window means holding a lock across arbitrary host code, trading a
benign race for a deadlock reachable from any callback.
Narrowed to what holds. One effect is not self-limiting and is now ordered: a
checkpoint save. Cancellation cannot stop it — a
CheckpointStoremay ignorectx, and the built-in one documents that it does, so the cancelled contextClose publishes is precisely the signal that store is specified to ignore. A
A
durableGatenarrows it. The failure is sequential, not adversarial —Close, then a second connector over the same store, then a late save landing on
a lineage the new run already loaded.
Stated precisely, because an earlier version of this paragraph claimed more than
the gate delivers: the guarantee is that no save commences after Close
returns, where commencing means claiming the gate — an act that is atomic
against Close and takes no host code with it. Close does not wait, and must
not: holding the lock across
CheckpointStore.Saveself-deadlocks a store whoseSavecallsClose(documented callable from anywhere), and a store that merelystalls would block every
Closefor as long as it stalls.So a save that claimed the gate and was then descheduled can still land after a
replacement connector loaded — the window is narrowed from [decision, write] to
[claim, write], not closed. That residual moves a checkpoint backward against
§23's "checkpoints only move forward"; its cost is bounded replay, not skipped
events. Closing it needs
Closeto wait (rejected above) or a fencing token onCheckpointStore— a six-SDK contract change. Tracked in #784.Released before the observer callbacks, because Close-from-a-callback is
documented as supported and holding the gate across host code deadlocks
Observer.Checkpointagainst the very save that fired it — verified by mutation.Also: a cancelled checkpoint load is no longer
Terminal(checkpoint_load).Blocker 7 — observers see origins only
#777's redactor applied to
Observer.Gapand bothCatchUpStartedsites. Anaccepted 410 latches the server's resume URL as reconnect state, so the
reconnect announces its walk carrying it — redacting
Gapalone would have leftthe identical URL leaving through a different callback one reconnect later.
Observer.Disconnectedis redacted, and this paragraph used to say theopposite. The original reasoning — an error is opaque text, and stripping a
credential out of arbitrary text means modelling the credential, which §23's
"opaque bearer" contract forbids — argued for leaving it alone. A later round
found a ticket reaching the callback through the raw seam read error, which no
seam obligation can repair from the connector's side, so both arguments now go
through closed vocabularies:
observableDisconnectReasonmaps everyunrecognized peer reason to
"other", andobservableSocketErrorreduces everycause to one the connector owns, degrading anything unrecognized to
errSocketFailed."Reduces" is load-bearing and was the last hole: matching a sentinel is not the
same as being one, so a seam returning
fmt.Errorf("read %s: %w", cableURL, context.Canceled)matched a recognized arm while its text carried the ticket.Every arm now returns the connector's own value rather than its argument;
errors.Isstill matches, and the wrapper does not survive.The seam obligation remains on
Dial,ReadFrameandWriteFrame— it is whatkeeps a preserved typed error safe — but the connector no longer depends on it
being honored.
#763 — staleness arms at socket open
Observer.Connectedfired between the socket opening and the window thatmeasures silence on it. Whatever a host's callback spent was time the window
never counted. Observed from inside the callback, which is the only place the
ordering is visible.
#760 — an occupied deferral slot no longer blinds the drain's fatal scan
drainScanreturned the moment it found the slot occupied, on the reasoningthat everything queued "arrived behind this one". That is a claim about arrival
order, where §23's carve-out is a claim about which verdict governs. With
any non-fatal outcome parked ahead of it the scan looked at nothing — and the
budget it runs under is
pumpDepth+1, sized indrain's own comment to reachevery frame the pump had already read. An occupied slot spent none of it.
The planned fix was a bounded deferral queue carved out of
pumpDepth; thatturned out to be unnecessary. Only one deferral is ever dispatched —
pumpExited,dispatchDisconnectand the invalid-frame teardown all end thecycle — so a queue would be a buffer sized for a delivery that cannot happen.
The scan keeps the first outcome and discards the rest as it passes them, which
removes the capacity question entirely: no share of
pumpDepth, no channelresize, no change to the published memory bound, no fixture sweep.
The new subtest needs both halves of the two around it: deferring during the
entry poll occupies the slot, and queuing the fatal frame mid-drain puts it
where only the scan can find it. Queued earlier, the ownership cut consumes it
first — which is how the first draft passed against un-fixed code.
On #758/#759: I could not reproduce the missing-wake-source hang. Every path
leaves a wake. The overshoot is real (close to two staleness windows) and
provably cannot exceed two, so the bound is documented in place rather than
patched, and carried to bc3 as an open §23 contract question.
Verification
Pristine worktree, one pass, clean tree before and after:
go build/go vet/-race -count=1/-count=5— passmake go-lint0 issues;gosecon the CI-pinned v2.23.0 (hash-verified) 0 issuesmake check— exit 0TestWalkFailureBetweenPagesboth subtests — the invariant that killed thereviewers' one-liner survives
-race: 0 failures, 0 data races, re-earnedbecause this PR rewrites the files that wait
Kill-matrix correction
Row 15's claim was inherited from the family README and is wrong; #778
corrects it. Tier 2 cannot prove zero egress to a foreign redirect target,
because the driver is the seam and manufactures the verdict. That is a
Layer-1 property, tracked for G1b.