feat(replication): warn-level visibility for parked/declined blob sends

[kriszyp]

Summary

sendBlobs() in replication/replicationConnection.ts had two silent paths that only logged at debug (production runs at warn):

1. A send parked on the outstanding-sends concurrency cap (MAX_OUTSTANDING_BLOBS_BEING_SENT) — invisible if a stuck predecessor send saturates the cap and starves successors.
2. A send declined outright — socket already closed, or a shutdown drain in progress (checked both before and after the concurrency-cap wait).

Both are now throttled logger.warn lines with fileId, database, node, and reason, so a starved/declined blob send is diagnosable from default logs.

Motivation

Live incident (internal demo.reflex cluster, 5.1.19): a deploy's 26.7MB blob payload never went out to the peer for 60+ seconds while the connection was healthy and no drain was active; the peer's 20s blobReadTimeout failed the deploy. Every plausible cause (parked on the cap, silent decline, starvation) only logged at debug, so post-hoc diagnosis was impossible. The trigger is rare and unreproducible, so the next occurrence needs to self-diagnose from default logs.

What changed

• New replication/blobSendWarnThrottle.ts: pure, testable throttle/aggregate helpers, following the same throttled-suppression pattern already used for blob send errors (blobSendErrorsSuppressed/lastBlobSendErrorLog a few lines below). Three independent throttle buckets (park, unpark, decline) so a burst under one cause can't drown out the others.
• sendBlobs():
  • Sets a PARK_WARN_MS-delay timer (default 5000ms, env HARPER_BLOB_SEND_PARK_WARN_MS, matching blobSendDrain.ts's HARPER_BLOB_SEND_* pattern) the moment a send is about to park on the cap. It only fires — and only warns — if the send is still parked once the threshold elapses; a send that acquires a slot quickly never logs anything.
  • Warns once more when a send that had warned finally unparks, with total waited time.
  • Promotes the three decline paths (wsClosed pre-queue, isDrainingBlobSends() pre-queue, isDrainingBlobSends() post-queue) from silent/debug to a throttled warn stating the blob will rely on peer re-request.
• Unit tests for the throttle/aggregation logic (unitTests/replication/blobSendWarnThrottle.test.mjs), matching blobSendDrain.test.mjs's pure-function testing style.

No changes to the per-chunk hot path (noteBlobSendProgress untouched) or to send behavior/control-flow — only timers and warn calls on the already-slow park/decline paths.

Where to look

• replication/replicationConnection.ts — the sendBlobs() diff (search parkWarn/declineWarn) is the load-bearing change; the rest of the function is untouched.
• replication/blobSendWarnThrottle.ts — the throttle logic itself.

Notes for the reviewer

• This is the only change intended for the next v5.1.x patch (kept self-contained/cherry-pickable): it's a diagnostic lens for a rare, unreproduced field failure (deploy blob content not sent for 60+s on a healthy connection), so the next occurrence self-diagnoses from default warn logs. The behavioral fix (fix(deploy): retry peer-side payload_blob read on transient 503 stall harper#1838, deploy-payload retry-on-503) stays on main/5.2 for now to keep patch complexity down; its 5.1 fate gets decided after diagnosis.
• Follow-up Remove (or dial down) blob-send diagnostic warn logging once send-starvation is diagnosed #592 tracks removing/demoting this instrumentation once the starvation is root-caused — it's diagnostic, not permanent complexity.
• harper-pro#527 (receiver re-request on close) needs nothing: it's already on v5.1 (rode in with cherry-pick: feat(replication): gracefully drain in-flight blob sends before worker shutdown (#529 → v5.1) #552, commit 96fb029, since v5.1.18).

🤖 Generated with Claude Code

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Code Review

This pull request introduces a throttled-warning mechanism for blob-send operations to improve observability of starvation and decline scenarios. It adds a new utility for managing throttle states and integrates it into the replication connection logic to log warnings when sends are parked on concurrency caps or declined during shutdown. The reviewer identified an issue with the environment variable parsing for PARK_WARN_MS, where setting the value to 0 incorrectly falls back to the default of 5000 due to falsy evaluation, and provided a robust fix.

[github-actions]

Patch cherry-pick: merged

Cherry-picked onto v5.1.