Skip to content

Fix Sentinel pool capacity loss after failover - #4193

Merged
petyaslavova merged 3 commits into
redis:masterfrom
Sanjays2402:fix/sentinel-pool-capacity
Jul 17, 2026
Merged

Fix Sentinel pool capacity loss after failover#4193
petyaslavova merged 3 commits into
redis:masterfrom
Sanjays2402:fix/sentinel-pool-capacity

Conversation

@Sanjays2402

@Sanjays2402 Sanjays2402 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Description of change

Fixes #4187. During a Sentinel master failover, the sync pool can reject an in-flight connection it originally created, but releasing it did not restore _created_connections, permanently consuming capacity. The pool now reclaims that slot only for connections created in its current process, and the regression test fails before the fix and passes after it.

Pull Request check-list

  • Do tests and lints pass with this change?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)? (Pending upstream CI.)
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)? (Not applicable; no API change.)
  • Is there an example added to the examples folder (if applicable)? (Not applicable.)

NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.


Note

Low Risk
Small, targeted change to sync ConnectionPool.release() for the non-owned-connection path; guarded by PID check to avoid incorrect decrements after fork.

Overview
Fixes permanent connection pool shrink after Sentinel master failover (#4187). When SentinelConnectionPool rejects a connection on release() because it no longer points at the current master, the pool now decrements _created_connections (guarded by connection.pid == self.pid) so the slot is not lost forever.

Adds test_master_failover_reclaims_discarded_connection_slot, which simulates failover between two master addresses and asserts the pool can still allocate connections after repeated get/release cycles.

Reviewed by Cursor Bugbot for commit 72581ec. Bugbot is set up for automated code reviews on this repo. Configure here.

Sentinel master failovers can make an in-flight connection unowned, causing the pool to discard it without restoring capacity. Reclaim the slot only when the connection was created in the current pool process, preserving fork safety. Add a regression test covering repeated failover discards.

@eeshsaxena eeshsaxena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and this one is worth taking - the pool ends up permanently unusable, not just leaky. I reproduced it on current master with your scenario:

_created_connections after 2 failover releases: 2  (expected 0)
get_connection() -> RAISED: MaxConnectionsError Too many connections

So after max_connections in-flight connections happen to span a failover, the pool is exhausted forever even though it holds no connections.

The mechanism matches your description. ConnectionPool.owns_connection() is just connection.pid == self.pid (connection.py:3316), while SentinelConnectionPool.owns_connection() is address_matches and super().owns_connection(...) (sentinel.py:190). So release()'s else branch is reachable for two different reasons:

  1. the connection genuinely came from another process/pool (pid differs) - the count must not be touched, which is what the old comment was protecting;
  2. this pool created the connection but the subclass rejected it because the master address moved - here the slot was counted in make_connection() and must be given back.

Gating the decrement on connection.pid == self.pid is exactly the predicate that separates (2) from (1), so the fix looks correct to me, and the old comment's intent is preserved for case (1).

One small suggestion: connection.pid == self.pid is literally the body of the base ConnectionPool.owns_connection(). Since self.owns_connection is overridden here, you can't call it, but ConnectionPool.owns_connection(self, connection) would express the intent directly and keep the two in sync if the base ownership rule ever changes. Purely cosmetic.

For scope: the asyncio pool tracks capacity with _in_use_connections / _available_connections rather than a _created_connections counter, so it doesn't look affected by this particular leak.

The regression test is well targeted - it fails on master with MaxConnectionsError and passes with the fix. Deferring to the maintainers.

@petyaslavova petyaslavova left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Sanjays2402, thank you for your contribution! Two minor changes need to be handled, and it will be ready for merging.

Comment thread redis/connection.py Outdated
Comment thread tests/test_sentinel.py

@petyaslavova petyaslavova left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@petyaslavova
petyaslavova merged commit 614df57 into redis:master Jul 17, 2026
368 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SentinelConnectionPool permanently loses a pool slot on every master switch -> "Too many connections" forever

3 participants