Feat/Add graceful shutdown support for grpc server #53
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yeong-hwan
commented
Jul 3, 2025
Comment on lines
+53
to
+61
| def _setup_signal_handlers(self): | ||
| """Setup signal handlers (inspired by Gunicorn arbiter.py)""" | ||
| # Store SIGTERM handler | ||
| self._original_sigterm_handler = signal.signal(signal.SIGTERM, self._handle_sigterm) | ||
|
|
||
| # Also set SIGINT handler (Ctrl+C) | ||
| signal.signal(signal.SIGINT, self._handle_sigterm) | ||
|
|
||
| self.stdout.write("Signal handlers registered for graceful shutdown") |
Contributor
Author
There was a problem hiding this comment.
Added _setup_signal_handlers() method to register signal handlers
Comment on lines
+63
to
+66
| def _handle_sigterm(self, signum, frame): | ||
| """Handle SIGTERM signal to start graceful shutdown""" | ||
| self.stdout.write(f"Received signal {signum}. Starting graceful shutdown...") | ||
| self._shutdown_event.set() |
Contributor
Author
There was a problem hiding this comment.
Added _handle_sigterm() method to handle shutdown signals
Sets the shutdown event when SIGTERM or SIGINT is received, triggering the graceful shutdown process.
Comment on lines
+68
to
+88
| def _graceful_shutdown(self, server): | ||
| """Gracefully shutdown the server""" | ||
| try: | ||
| # Stop accepting new connections | ||
| self.stdout.write("Stopping server from accepting new connections...") | ||
|
|
||
| # Stop gRPC server (with grace=True to wait for ongoing requests to complete) | ||
| if hasattr(server, 'stop'): | ||
| # For synchronous server | ||
| server.stop(grace=True) | ||
| else: | ||
| # For asynchronous server | ||
| asyncio.create_task(server.stop(grace=True)) | ||
|
|
||
| # Send Django signal | ||
| grpc_shutdown.send(None) | ||
|
|
||
| self.stdout.write("Graceful shutdown completed") | ||
|
|
||
| except Exception as e: | ||
| self.stderr.write(f"Error during graceful shutdown: {e}") |
Contributor
Author
There was a problem hiding this comment.
Added _graceful_shutdown() and _graceful_shutdown_async() methods for server cleanup
Handles the actual server shutdown process, stopping new connections, waiting for ongoing requests to complete, and sending Django signals.
Comment on lines
107
to
+148
| def _serve(self, max_workers, port, *args, **kwargs): | ||
| """ | ||
| Run gRPC server | ||
| """ | ||
| autoreload.raise_last_exception() | ||
| self.stdout.write("gRPC server starting at %s" % datetime.datetime.now()) | ||
|
|
||
| # Only setup signal handlers when not in autoreload mode | ||
| # autoreload runs in a separate thread, not the main thread, so signal handlers cannot be registered | ||
| if not kwargs.get("autoreload", False): | ||
| self._setup_signal_handlers() | ||
|
|
||
| server = create_server(max_workers, port) | ||
| self._server = server | ||
|
|
||
| server.start() | ||
|
|
||
| self.stdout.write("gRPC server is listening port %s" % port) | ||
|
|
||
| if kwargs["list_handlers"] is True: | ||
| # Print handler list if list_handlers option is enabled (default: False) | ||
| if kwargs.get("list_handlers", False): | ||
| self.stdout.write("Registered handlers:") | ||
| for handler in extract_handlers(server): | ||
| self.stdout.write("* %s" % handler) | ||
|
|
||
| server.wait_for_termination() | ||
| # Send shutdown signal to all connected receivers | ||
| grpc_shutdown.send(None) | ||
| # Only execute graceful shutdown logic when not in autoreload mode | ||
| if not kwargs.get("autoreload", False): | ||
| # Wait loop for graceful shutdown | ||
| try: | ||
| while not self._shutdown_event.is_set(): | ||
| time.sleep(0.1) | ||
| except KeyboardInterrupt: | ||
| self.stdout.write("Received keyboard interrupt, starting graceful shutdown...") | ||
| self._shutdown_event.set() | ||
|
|
||
| # Perform graceful shutdown | ||
| self._graceful_shutdown(server) | ||
| else: | ||
| # Use original wait_for_termination for autoreload mode | ||
| server.wait_for_termination() | ||
| # Send shutdown signal to all connected receivers | ||
| grpc_shutdown.send(None) |
Contributor
Author
There was a problem hiding this comment.
Modified _serve() and _serve_async() methods to include graceful shutdown logic
Integrates signal handling and graceful shutdown logic into the main server serving methods
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds graceful shutdown support to the Django gRPC server, inspired by Gunicorn's arbiter.py implementation. The feature addresses the PID 1 problem commonly encountered in containerized environments and ensures that the server can be safely terminated while allowing ongoing requests to complete.
Problem Statement
In containerized environments, especially Kubernetes, applications often run as PID 1 inside containers.
When PID 1 processes don't handle signals properly, they can:
Why gRPC servers deployed on Kubernetes may not respond to SIGTERM:
What does this PR do?
grace=True)grpc_shutdownsignal to all connected receiversKey Features
Testing
The implementation includes: