fix(sentinel): back off between failed topology refreshes - #1017
Open
FZambia wants to merge 1 commit into
Open
Conversation
refreshRetry was an unbounded 'goto retry' loop with no delay. When every sentinel is unreachable — which is what happens during a failover where the primary and a co-located sentinel go down together — refresh() fails immediately and the loop spins as fast as the CPU allows, dialing every sentinel in the list on each iteration. Apply equal-jitter exponential backoff between attempts and bail out once Close() has been called, so a client shut down while its sentinels are unreachable does not leak the goroutine. The base grows to 512ms, one shift below the 1024ms cap, so base+jitter always lands under the cap rather than being clamped to it. Capping the sum at the same magnitude as the base would make every sample come out at exactly the cap once the base reached it: the jitter would vanish at steady state and a fleet of clients riding out a long outage would retry in lockstep — the thundering herd the equal-jitter scheme is there to prevent.
FZambia
force-pushed
the
fix/sentinel-refresh-backoff
branch
from
August 2, 2026 07:00
a6938cc to
b2a0dd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
refreshRetryretried failed topology refreshes with no delay and never exited onClose. This adds equal-jitter backoff and a prompt shutdown.Impact
refreshRetryspins as fast as the CPU allows, dialing every sentinel on each iteration — one client can saturate a core and flood the surviving sentinels with connection attempts exactly while they are running the election.stop, so it retried forever.Root cause
refreshRetrywas an unboundedgoto retryloop with no delay and no shutdown check.Fix
Equal-jitter exponential backoff between attempts (1ms base, 1s cap), and return once
Close()has been called. Still retries indefinitely — only the spin is removed.Tests
TestRefreshRetryDelay— the delay is always positive, never exceeds the cap, and grows with attempts.TestSentinelRefreshRetryBackoff— pins both properties under the leak detector: no spin (≤ a few dozen attempts in 200ms, vs hundreds of thousands before) and prompt exit onClose.Note
Medium Risk
Touches sentinel failover/reconnect behavior under total sentinel outage; mis-tuned backoff could slow recovery, but changes are localized and heavily tested.
Overview
Replaces the sentinel client’s
refreshRetrytight retry loop with equal-jitter exponential backoff between failed topology refreshes, so unreachable sentinels during failover are not hammered with CPU-bound dial storms.refreshRetrynow loops whilec.stopis clear: on failure it sleeps usingrefreshRetryDelay(ms-scale base, 1s cap, jitter preserved at the cap) and returns on success orClose().waitBeforeRetrybreaks out of long sleeps in 20ms steps whenstopis set.New tests lock in delay bounds/growth/jitter, interruptible wait on close, and that failed refresh no longer spins while the goroutine exits after
Close().Reviewed by Cursor Bugbot for commit b2a0dd1. Bugbot is set up for automated code reviews on this repo. Configure here.