Skip to content

asgi: decrement nr_conns on server-initiated connection close (#3661) - #3664

Closed
apoorvdarshan wants to merge 1 commit into
benoitc:masterfrom
apoorvdarshan:fix-3661-asgi-nr-conns-leak
Closed

asgi: decrement nr_conns on server-initiated connection close (#3661)#3664
apoorvdarshan wants to merge 1 commit into
benoitc:masterfrom
apoorvdarshan:fix-3661-asgi-nr-conns-leak

Conversation

@apoorvdarshan

Copy link
Copy Markdown
Contributor

Summary

Fixes #3661. On the ASGI worker, nr_conns is never decremented for server-initiated connection closes, so it grows with worker age and every graceful stop (SIGTERM, --max-requests autorestart) blocks for the full graceful_timeout — even on an idle worker. This hits any setup with Connection: close / HTTP/1.0-per-request (e.g. nginx proxying to the worker).

Root cause

ASGIProtocol used a single _closed flag for two different jobs:

  • _close_transport() sets _closed = True to mark the transport closed;
  • connection_lost() used if self._closed: return as its idempotency guard.

A client-initiated close calls connection_lost() directly (_closed is False) → the count is decremented. A server-initiated close runs _close_transport() first, setting _closed = True; when asyncio then calls connection_lost(), the guard returns early and skips self.worker.nr_conns -= 1 (and the rest of the cleanup).

Fix

Introduce a dedicated _conn_lost_handled flag for connection_lost()'s idempotency guard, independent of the transport's _closed state. The cleanup (decrementing nr_conns, disconnect signalling, task cancellation) now runs exactly once regardless of who initiated the close.

Tests

  • Added test_server_initiated_close_still_decrements_nr_conns: _close_transport() then connection_lost() must leave nr_conns == 0. It fails on master (stays 1) and passes here.
  • tests/test_asgi_disconnect.py (14) and tests/test_asgi_worker.py (32) pass; the full ASGI suite is green (628 passed, 121 skipped); pycodestyle clean.

Happy to add a changelog entry if you'd like one.

Disclosure: prepared with AI assistance; reviewed and verified locally.

The ASGI protocol shared one _closed flag for two purposes: marking the
transport closed (_close_transport) and guarding the connection_lost()
cleanup. On a server-initiated close (Connection: close, keepalive timeout,
etc.) _close_transport() set _closed=True before asyncio reported the
transport loss, so connection_lost() returned early and never decremented
worker.nr_conns. The count grew with worker age and every graceful stop
waited out the full graceful_timeout.

Use a dedicated _conn_lost_handled flag for the connection_lost() idempotency
guard so its cleanup (nr_conns decrement, disconnect signalling) always runs
exactly once regardless of who initiated the close.

Fixes benoitc#3661.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@benoitc

benoitc commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Reopening to trigger CI, which never fired on this PR. Thanks for the fix, the approach here is the right one.

@benoitc

benoitc commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Thanks, your analysis and fix were the right ones. Superseded by #3686, which carries your commit over unchanged and still under your authorship, and adds coverage for three gaps: the keepalive close path leaking as well, the cleanup steps the early return also skipped (keepalive timer, reader EOF, body receiver disconnect signal), and the reported symptom itself, that _shutdown() burns the full graceful_timeout.

Closing here in favour of #3686.

@benoitc benoitc closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ASGI worker: nr_conns never decremented on server-initiated close — every graceful stop waits out the full graceful_timeout

2 participants