Skip to content

fix(sentinel): back off between failed topology refreshes - #1017

Open
FZambia wants to merge 1 commit into
redis:mainfrom
FZambia:fix/sentinel-refresh-backoff
Open

fix(sentinel): back off between failed topology refreshes#1017
FZambia wants to merge 1 commit into
redis:mainfrom
FZambia:fix/sentinel-refresh-backoff

Conversation

@FZambia

@FZambia FZambia commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

refreshRetry retried failed topology refreshes with no delay and never exited on Close. This adds equal-jitter backoff and a prompt shutdown.

Impact

  • When every sentinel is unreachable (a failover that takes the primary and a co-located sentinel down together), refreshRetry spins as fast as the CPU allows, dialing every sentinel on each iteration — one client can saturate a core and flood the surviving sentinels with connection attempts exactly while they are running the election.
  • A client closed during this leaks the goroutine: the loop never checked stop, so it retried forever.

Root cause

refreshRetry was an unbounded goto retry loop with no delay and no shutdown check.

Fix

Equal-jitter exponential backoff between attempts (1ms base, 1s cap), and return once Close() has been called. Still retries indefinitely — only the spin is removed.

Tests

  • TestRefreshRetryDelay — the delay is always positive, never exceeds the cap, and grows with attempts.
  • TestSentinelRefreshRetryBackoff — pins both properties under the leak detector: no spin (≤ a few dozen attempts in 200ms, vs hundreds of thousands before) and prompt exit on Close.

Note

Medium Risk
Touches sentinel failover/reconnect behavior under total sentinel outage; mis-tuned backoff could slow recovery, but changes are localized and heavily tested.

Overview
Replaces the sentinel client’s refreshRetry tight retry loop with equal-jitter exponential backoff between failed topology refreshes, so unreachable sentinels during failover are not hammered with CPU-bound dial storms.

refreshRetry now loops while c.stop is clear: on failure it sleeps using refreshRetryDelay (ms-scale base, 1s cap, jitter preserved at the cap) and returns on success or Close(). waitBeforeRetry breaks out of long sleeps in 20ms steps when stop is set.

New tests lock in delay bounds/growth/jitter, interruptible wait on close, and that failed refresh no longer spins while the goroutine exits after Close().

Reviewed by Cursor Bugbot for commit b2a0dd1. Bugbot is set up for automated code reviews on this repo. Configure here.

refreshRetry was an unbounded 'goto retry' loop with no delay. When every
sentinel is unreachable — which is what happens during a failover where the
primary and a co-located sentinel go down together — refresh() fails
immediately and the loop spins as fast as the CPU allows, dialing every sentinel
in the list on each iteration.

Apply equal-jitter exponential backoff between attempts and bail out once
Close() has been called, so a client shut down while its sentinels are
unreachable does not leak the goroutine.

The base grows to 512ms, one shift below the 1024ms cap, so base+jitter always
lands under the cap rather than being clamped to it. Capping the sum at the same
magnitude as the base would make every sample come out at exactly the cap once
the base reached it: the jitter would vanish at steady state and a fleet of
clients riding out a long outage would retry in lockstep — the thundering herd
the equal-jitter scheme is there to prevent.
@FZambia
FZambia force-pushed the fix/sentinel-refresh-backoff branch from a6938cc to b2a0dd1 Compare August 2, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant