asgi: decrement nr_conns on server-initiated connection close (#3661) - #3664
Closed
apoorvdarshan wants to merge 1 commit into
Closed
asgi: decrement nr_conns on server-initiated connection close (#3661)#3664apoorvdarshan wants to merge 1 commit into
apoorvdarshan wants to merge 1 commit into
Conversation
The ASGI protocol shared one _closed flag for two purposes: marking the transport closed (_close_transport) and guarding the connection_lost() cleanup. On a server-initiated close (Connection: close, keepalive timeout, etc.) _close_transport() set _closed=True before asyncio reported the transport loss, so connection_lost() returned early and never decremented worker.nr_conns. The count grew with worker age and every graceful stop waited out the full graceful_timeout. Use a dedicated _conn_lost_handled flag for the connection_lost() idempotency guard so its cleanup (nr_conns decrement, disconnect signalling) always runs exactly once regardless of who initiated the close. Fixes benoitc#3661.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Owner
|
Reopening to trigger CI, which never fired on this PR. Thanks for the fix, the approach here is the right one. |
Owner
|
Thanks, your analysis and fix were the right ones. Superseded by #3686, which carries your commit over unchanged and still under your authorship, and adds coverage for three gaps: the keepalive close path leaking as well, the cleanup steps the early return also skipped (keepalive timer, reader EOF, body receiver disconnect signal), and the reported symptom itself, that Closing here in favour of #3686. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3661. On the ASGI worker,
nr_connsis never decremented for server-initiated connection closes, so it grows with worker age and every graceful stop (SIGTERM,--max-requestsautorestart) blocks for the fullgraceful_timeout— even on an idle worker. This hits any setup withConnection: close/ HTTP/1.0-per-request (e.g. nginx proxying to the worker).Root cause
ASGIProtocolused a single_closedflag for two different jobs:_close_transport()sets_closed = Trueto mark the transport closed;connection_lost()usedif self._closed: returnas its idempotency guard.A client-initiated close calls
connection_lost()directly (_closedisFalse) → the count is decremented. A server-initiated close runs_close_transport()first, setting_closed = True; when asyncio then callsconnection_lost(), the guard returns early and skipsself.worker.nr_conns -= 1(and the rest of the cleanup).Fix
Introduce a dedicated
_conn_lost_handledflag forconnection_lost()'s idempotency guard, independent of the transport's_closedstate. The cleanup (decrementingnr_conns, disconnect signalling, task cancellation) now runs exactly once regardless of who initiated the close.Tests
test_server_initiated_close_still_decrements_nr_conns:_close_transport()thenconnection_lost()must leavenr_conns == 0. It fails onmaster(stays1) and passes here.tests/test_asgi_disconnect.py(14) andtests/test_asgi_worker.py(32) pass; the full ASGI suite is green (628 passed, 121 skipped);pycodestyleclean.Happy to add a changelog entry if you'd like one.
Disclosure: prepared with AI assistance; reviewed and verified locally.