Skip to content

fix accepting one more request after worker shutdown command is sent - #2

Closed
shanet wants to merge 3 commits into
mainfrom
fix_accept_after_worker_shutdown
Closed

shanet wants to merge 3 commits into
mainfrom
fix_accept_after_worker_shutdown

Conversation

@shanet

@shanet shanet commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Description

My team noticed an issue when a TERM signal is sent to shut down a worker, it's possible for that worker to accept & process one more request before shutting down. This is when queue_requests false and drain_on_shutdown false are configured.

The potential shutdown sequence is:

  1. Request A occupies the worker's only request thread.
  2. A connection for Request B becomes pending. The listener thread waits for thread-pool capacity.
  3. Request A sends TERM to its worker. Puma's signal handler writes STOP to the listener's control pipe.
  4. Request A returns a 504 and releases capacity.
  5. The listener stops waiting and accepts the connection for Request B without checking for a pending STOP command first. Processing STOP afterwards does not prevent B from running.

This happens in https://github.com/aha-app/puma/blob/main/lib/puma/server.rb#L402 where if a TERM signal is sent while the worker is waiting for capacity in pool.wait_until_not_full it will accept a subsequent request and process it.

A test is included to reproduce this sequence.

The fix here being fairly straightforward: Check again for a stop message after pool.wait_until_not_full returns. This prevents another request from being processed and shuts down the worker immediately.

Why this matters: In our case, we are using the shutdown behavior to stop processing requests by a worker in a process address space that we can no longer trust not to leak client data between requests due to errors during processing of the request. Thus, processing even one more request after a TERM is sent may seem trivial, but is potentially an (unlikely) attack vector that we need to guard against.

Your checklist for this pull request

  • I have reviewed the guidelines for contributing to this repository.
  • I have added (or updated) appropriate tests if this PR fixes a bug or adds a feature.
  • My pull request is 100 lines added/removed or less so that it can be easily reviewed.
  • If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
  • If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
  • I have updated the documentation accordingly.
  • All new and existing tests passed, including Rubocop.

@shanet
shanet force-pushed the fix_accept_after_worker_shutdown branch from b76da9f to f255f97 Compare September 29, 2026 22:25
@shanet
shanet force-pushed the fix_accept_after_worker_shutdown branch from f255f97 to 0a4a376 Compare September 29, 2026 22:36
@shanet shanet closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant