Fix Sentinel pool capacity loss after failover - #4193
Conversation
Sentinel master failovers can make an in-flight connection unowned, causing the pool to discard it without restoring capacity. Reclaim the slot only when the connection was created in the current pool process, preserving fork safety. Add a regression test covering repeated failover discards.
eeshsaxena
left a comment
There was a problem hiding this comment.
Confirmed, and this one is worth taking - the pool ends up permanently unusable, not just leaky. I reproduced it on current master with your scenario:
_created_connections after 2 failover releases: 2 (expected 0)
get_connection() -> RAISED: MaxConnectionsError Too many connections
So after max_connections in-flight connections happen to span a failover, the pool is exhausted forever even though it holds no connections.
The mechanism matches your description. ConnectionPool.owns_connection() is just connection.pid == self.pid (connection.py:3316), while SentinelConnectionPool.owns_connection() is address_matches and super().owns_connection(...) (sentinel.py:190). So release()'s else branch is reachable for two different reasons:
- the connection genuinely came from another process/pool (pid differs) - the count must not be touched, which is what the old comment was protecting;
- this pool created the connection but the subclass rejected it because the master address moved - here the slot was counted in
make_connection()and must be given back.
Gating the decrement on connection.pid == self.pid is exactly the predicate that separates (2) from (1), so the fix looks correct to me, and the old comment's intent is preserved for case (1).
One small suggestion: connection.pid == self.pid is literally the body of the base ConnectionPool.owns_connection(). Since self.owns_connection is overridden here, you can't call it, but ConnectionPool.owns_connection(self, connection) would express the intent directly and keep the two in sync if the base ownership rule ever changes. Purely cosmetic.
For scope: the asyncio pool tracks capacity with _in_use_connections / _available_connections rather than a _created_connections counter, so it doesn't look affected by this particular leak.
The regression test is well targeted - it fails on master with MaxConnectionsError and passes with the fix. Deferring to the maintainers.
petyaslavova
left a comment
There was a problem hiding this comment.
Hey @Sanjays2402, thank you for your contribution! Two minor changes need to be handled, and it will be ready for merging.
Description of change
Fixes #4187. During a Sentinel master failover, the sync pool can reject an in-flight connection it originally created, but releasing it did not restore
_created_connections, permanently consuming capacity. The pool now reclaims that slot only for connections created in its current process, and the regression test fails before the fix and passes after it.Pull Request check-list
NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.
Note
Low Risk
Small, targeted change to sync
ConnectionPool.release()for the non-owned-connection path; guarded by PID check to avoid incorrect decrements after fork.Overview
Fixes permanent connection pool shrink after Sentinel master failover (#4187). When
SentinelConnectionPoolrejects a connection onrelease()because it no longer points at the current master, the pool now decrements_created_connections(guarded byconnection.pid == self.pid) so the slot is not lost forever.Adds
test_master_failover_reclaims_discarded_connection_slot, which simulates failover between two master addresses and asserts the pool can still allocate connections after repeated get/release cycles.Reviewed by Cursor Bugbot for commit 72581ec. Bugbot is set up for automated code reviews on this repo. Configure here.