fix(health): keep serving probes after a dropped connection - #117
fix(health): keep serving probes after a dropped connection#117cursor[bot] wants to merge 15 commits into
Conversation
RED: the domain health snapshot is not yet exposed as GET /live and GET /ready with fail-closed readiness and a safe as-built OpenAPI 3.2 contract.
Map RuntimeHealthSnapshot to GET /live and GET /ready with fail-closed readiness, RFC 9457 problem details for unsupported requests, and an as-built OpenAPI 3.2.0 document that lists only those operations.
Exercise HealthHttpResponse::content_type and GET /ready?capability= from the library test module so Linux line coverage can reach the remaining unexecuted instantiations.
* feat(health): bind a TCP listener for operator probes GET /live and GET /ready already exist as a request translator. Bind a blocking listener that serves one HTTP/1.1 request per accepted connection without adding public product routes, TLS, or SLO values. * docs(traceability): name Active PR #92 on the bound health listener * feat(health): answer probes from a PostgreSQL operational snapshot * test(health): require a composed PostgreSQL operational snapshot RED: runtime and relation probes exist, but callers still assemble liveness, backlog, integrity, and the postgres capability by hand. * feat(health): compose PostgreSQL probes into one operational snapshot Map runtime and relation probes plus caller-supplied backlog into a RuntimeHealthSnapshot. Probe failure becomes unknown/unready evidence and never returns a driver error. * feat(health): answer probes from a PostgreSQL operational snapshot Rebuild #95 on the current #92 listener plus the #93 snapshot composer. handle_postgres_health_http_request and accept_one_postgres_health_http observe the caller-owned store and reuse the existing probe translator. * docs(health): name #95 snapshot wiring on the current listener stack Restore the architecture mappings the rewrite dropped: TRACEABILITY module and capability rows, OPERABILITY fail-closed snapshot readiness, ADR-0014 as-built status, and the composed-snapshot changelog line. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
The inherited #72 recovery fixture inserted a processing consumption row without claim_deadline_at. Migration 0019 requires that column for processing rows, and the deadline trigger is UPDATE-only, so exact-head CI failed closed. Seed a valid persisted claim and assert the deadline survives COPY restore. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
…port RED: GET /live must not observe PostgreSQL, bare GET /ready must fail closed on a read-only store, incomplete or oversized requests must return 400 without echoing input, and the as-built OpenAPI /live operation must document HTTP 503. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Observe PostgreSQL only for GET /ready, after accept. Bare /ready requires postgres_operational_store. The listener applies a 2-second I/O timeout, rejects incomplete or oversized requests without echo, and adds Cache-Control: no-store plus Allow: GET on 405. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Operators can now run a blocking accept loop so a load balancer can keep asking GET /live and GET /ready. Interrupted accepts retry; any other accept, read, or write error stops the loop. TLS and keep-alive stay out of this slice. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Keep #91 as the predecessor translator head. Operators should land the serve-loop successor so a load balancer can keep probing. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
A load-balancer reset after accept no longer stops GET /live or GET /ready. Accept still retries Interrupted/ConnectionAborted/ConnectionReset and stops on WouldBlock. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Head 30515bd correctly splits accept from per-connection I/O. That is the #111 defect: a BrokenPipe/ConnectionReset after a successful accept must not stop later GET /live or GET /ready. Docs stay Active PR, not protected-main truth. Prefer this over #111, #91, and draft #102.
Do not merge this draft. Required checks are still queued. Independent last-push approval is still required. Do not self-approve.
Two evidence gaps remain on this exact head:
should_continue_after_serve_errorretriesConnectionReseton accept, but no test constructs that pair. ExistingConnectionResetassertions cover connection I/O andapply_request_readonly. The 100% owned-production branch gate can fail this or-arm.serve_postgres_health_httpinherits the shared loop, but the PostgreSQL listener contract only proves live-then-ready then stop. There is no drop-then-continue probe on that adapter.
Operator next action: keep #117 parked until those two contracts exist on an unchanged reviewed head. Close or retarget #111, #91, and #102 only after that landing head is integrated. Do not start HTTP session transport on this health stack; that waits on #109.
Sent by Cursor Automation: Fix Issues
| error.kind(), | ||
| io::ErrorKind::Interrupted | ||
| | io::ErrorKind::ConnectionAborted | ||
| | io::ErrorKind::ConnectionReset |
There was a problem hiding this comment.
io::ErrorKind::ConnectionReset is part of the accept-retry match, but serve_loop_retries_interrupted_accepts_and_stops_on_other_errors never constructs ServeIoSource::Accept + ConnectionReset. The later test uses ConnectionReset only with ServeIoSource::Connection, which is the unconditional true arm. Add the accept pair before treating branch coverage as proven.
| /// # Errors | ||
| /// | ||
| /// Returns the I/O error that stopped the loop. | ||
| pub fn serve_postgres_health_http( |
There was a problem hiding this comment.
This adapter claims a dropped probe does not stop later probes, but tests/postgres_health_http_contract.rs has no client-drop then later GET /live exchange. The in-memory serve_health_http listener test does not prove this handler path. Add that contract on the PostgreSQL serve loop.
|
Closing as a proven superseded predecessor. Fresh compare shows #117 head |
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
2 participants
Why
PR #111 at
9080ba1can serve until accept fails, but one dropped load-balancer probe stops the loop. A Kubernetes or proxy check that resets after GET/livethen leaves later probes unanswered.TDD
RED required
should_continue_after_serve_errorto keep running afterBrokenPipe/ConnectionReseton an accepted connection, and a listener contract that still answers GET/liveafter a client drops mid-exchange. GREEN splits accept from connection I/O. Interrupted, aborted, or reset accepts retry. Other accept errors, includingWouldBlock, still stop the loop.Scope
serve_health_http/serve_health_http_with/serve_postgres_health_httpkeep accepting after per-connection read or write errors.Out of scope
Test plan
cargo test --lib health --test health_http_listener_contract --test health_http_probe_contractcargo test --test documentation_architecture_contract --test traceability_active_pr_contractcargo clippy --all-targets -- -D warningsDo not merge until exact-head checks and independent last-push approval are satisfied. Never self-approve.
Operator next action
Call
serve_health_httporserve_postgres_health_httpafterbind_health_http. Point liveness at GET/liveand readiness at GET/ready. A dropped probe must not require a process restart. Close or retarget #111, #91, and #102 after this lands.