test(sync-service): reproduce FlushTracker stall when a tracked consumer dies without cleanup#4713
Open
balegas wants to merge 1 commit into
Open
test(sync-service): reproduce FlushTracker stall when a tracked consumer dies without cleanup#4713balegas wants to merge 1 commit into
balegas wants to merge 1 commit into
Conversation
…mer dies without cleanup Reproduces the long-standing "system stops flushing the WAL" stall: a consumer that dies without running terminate (untrappable :kill) after its shape is tracked in the FlushTracker leaves a permanently incomplete entry. Publish-time DOWN detection only fires for events routed to that same shape, and the ShapeCleaner path requires terminate to run — so if the shape's table goes quiet, the global flush boundary is pinned forever and confirmed_flush_lsn never advances for the whole stack, causing unbounded WAL growth on Postgres. The test proves the boundary stays pinned while every live consumer is fully flushed, and that explicitly removing the dead shape from the ShapeLogCollector immediately unblocks it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4713 +/- ##
=======================================
Coverage 60.04% 60.05%
=======================================
Files 397 397
Lines 43766 43766
Branches 12587 12585 -2
=======================================
+ Hits 26280 26283 +3
+ Misses 17405 17402 -3
Partials 81 81
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deterministic reproduction of the long-standing "system stops flushing the WAL" stall (replication slot
confirmed_flush_lsnstops advancing → unbounded WAL growth on Postgres). This PR adds a test only — no fix. The test passes because it asserts the current (buggy) behavior, documenting the stall.The stall
ShapeLogCollector'sFlushTrackertracks each delivered shape at commit until the consumer reports its storage flush. The global flush boundary is the minimum across incomplete entries, so a single entry that never completes pinsconfirmed_flush_lsnfor the entire stack.Existing cleanup paths for a dead consumer are:
DOWNdetection inConsumerRegistry.publish/2— but only when a later event is routed to that same shape.ShapeCleaner.handle_writer_termination/3from the consumer'sterminate/2— butterminatenever runs on an untrappable:kill(and is a no-op for:normal/:shutdown/{:shutdown, other}reasons).A consumer killed after its shape is tracked but before it flushes, on a shape whose table then goes quiet, escapes both paths: the FlushTracker entry is permanent and the WAL flush boundary never advances again — even though every live consumer is fully flushed.
The test
FlushTracker stall when tracked consumer dies without cleanup on a quiet shapeinshape_log_collector_test.exs:other_tableis killed with:killbefore flushingtest_table, so publish-time DOWN detection never fires; after the surviving consumer flushes it, no boundary update is emitted — the stallShapeLogCollector.remove_shape/2for the dead shape immediately unblocks the boundary to 50Fix direction (follow-up)
Reconcile FlushTracker entries against consumer liveness in the SLC (monitor consumer pids, or periodically sweep entries whose consumer is dead) instead of relying on publish-time detection and consumer self-reporting. A stall watchdog logging the blocking shape handles + consumer liveness would also make future occurrences diagnosable in production.
Test plan
mix test test/electric/replication/shape_log_collector_test.exs— 38 tests, 0 failures🤖 Generated with Claude Code