Make aspnetcore shutdown work by default on Kubernetes

[jkotalik]

Right now, as soon as an aspnetcore app sees CTRL + C, it immediately starts shutting down the server, rejecting and draining connections. This works well when running on command line, but when you move into k8s, requests may be dropped.

This article gives a general overview of the problem. Kubernetes, the ingress controller, CoreDNS need time to remove the IP address from their internal state.

There are some solutions where you could have a dotnet app listen for endpoint changes within the k8s cluster. The recommended approach is to actually just wait after receiving a SIGTERM signal, rather than immediately shutting down.

The modification would be to enable shutdown on k8s to:

1. Wait a bit longer before exiting
2. Continue to process incoming traffic, even though we get SIGTERM
3. Shutdown all connections (drain), especially long lived ones like DB or WebSockets
4. Close the process

A recommend time to wait is 15 seconds.

I think the most difficult part of this will be detecting if you are running inside of k8s. Besides that, a Task.Delay seems like an appropriate solution.

[davidfowl]

I don't think we should detect if you're in k8s, it should just be a setting that we recommend people set in k8s. That would be the least surprising thing to do. Likely something akin to this ShutdownTimeout or maybe we can repurpose it?

Should we throw if this setting is < the shutdown timeout? Are those 2 things configured in the same place? I think this should be implemented in the GenericWebHostedService.StopAsync.

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[halter73]

I don't think we should detect if you're in k8s, it should just be a setting that we recommend people set in k8s.

If we're going to require a specific action from developers to enable this this, why not just recommend doing the following as suggested in https://learnk8s.io/graceful-shutdown?

      lifecycle:
        preStop:
          exec:
            command: ["sleep", "15"]

From reading that article, this seems like a common problem. Do we know if any other frameworks continue processing new requests and accepting new connections after receiving a SIGTERM? Or if they even have the option?

[davidfowl]

If we're going to require a specific action from developers to enable this this, why not just recommend doing the following as suggested in https://learnk8s.io/graceful-shutdown?

We do right now.

From reading that article, this seems like a common problem. Do we know if any other frameworks continue processing new requests and accepting new connections after receiving a SIGTERM? Or if they even have the option?

Not sure, but I think we should build it in because it's simple and solves a common problem in a core scenario.

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[cdlliuy]

From reading that article, this seems like a common problem. Do we know if any other frameworks continue processing new requests and accepting new connections after receiving a SIGTERM? Or if they even have the option?

Not sure, but I think we should build it in because it's simple and solves a common problem in a core scenario.

per my knowledge of Golang, when the SIGTERM is triggered, the golang web server will stop to accept new connection, and
wait for checking the existing connection to be completed. If no active connection, the server will shutdown safely.

I think it will be helpful for aspnet customer if it can provide similar capacity. Currently, in our own application, we plan to add a customized middle ware to count the active http connection number (count ++ for incoming request, count-- when the request is done), and hold the applicationstopping handler until the connection number is 0

[davidfowl]

I think it will be helpful for aspnet customer if it can provide similar capacity. Currently, in our own application, we plan to add a customized middle ware to count the active http connection number (count ++ for incoming request, count-- when the request is done), and hold the applicationstopping handler until the connection number is 0

ASP.NET Core does this already. You can verify this by having a connection that does a Task.Delay for 10 seconds and watch the behavior on shutdown.

The above describes that we should introduce a small delay before we start the shutdown process. Which you can do today in 2 ways:

• Sleep in the ApplicationStopping event (kinda gross)
• Or use Make aspnetcore shutdown work by default on Kubernetes #30387 (comment)

[davidfowl]

Moving this into .NET 8 planning. We can make some small changes here to make this work a bit better than it does today in k8s (it can even be opt in).