ConsoleLifetime doesn't allow full graceful shutdown for ProcessExit

[davidmatson]

Describe the bug

When ConsoleLifetime shuts down via the AppDomain.ProcessExit path, it skips a number of steps done in normal graceful program termination.

Related to dotnet/extensions#1363, but perhaps the opposite problem (in this case, it shuts down too soon rather than hanging/too late).

To Reproduce

Program.cs:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using System;
using System.Threading;

class Program
{
    static int Main()
    {
        var builder = new HostBuilder()
            .ConfigureLogging(l =>
            {
                l.AddConsole();
                l.Services.AddSingleton<ILoggerProvider, CustomLoggerProvider>();
            }).ConfigureServices(s =>
            {
                s.AddSingleton<OtherService>();
            });

        using (IHost host = builder.Build())
        {
            host.Services.GetRequiredService<OtherService>();
            host.Start();
            // Edit: ignore Environment.Exit; instead use CTRL+C or docker stop here, per comments below.
            //Thread thread = new Thread(() => Environment.Exit(456));
            //thread.Start();
            host.WaitForShutdown();
            Console.WriteLine("Ran cleanup code inside using host block.");
        }

        Console.WriteLine("Ran cleanup code outside using host block.");
        Console.WriteLine("Returning exit code from Program.Main.");
        return 123;
    }

    class CustomLoggerProvider : ILoggerProvider
    {
        public ILogger CreateLogger(string categoryName)
        {
            return NullLogger.Instance;
        }

        public void Dispose()
        {
            Console.WriteLine("Ran logger cleanup code.");
        }
    }

    class OtherService : IDisposable
    {
        public void Dispose()
        {
            Console.WriteLine("Ran most service's cleanup code.");
        }
    }
}

App.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <SelfContained>false</SelfContained>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.2.0" />
  </ItemGroup>
</Project>

1. Run the code above as-is, and terminate with Ctrl+C. See all the cleanup code that runs.
2. Uncomment the two Thread lines above to let the program self-terminate via Process.Exit.

Expected behavior

The same cleanup code runs when exiting via ProcessExit as it did with Ctrl+C:

Ran cleanup code inside using host block.
Ran most service's cleanup code.
Ran logger cleanup code.
Ran cleanup code outside using host block.
Returning exit code from Program.Main.

X:\path\to\App.exe (process nnn) exited with code 123.

Actual behavior

Only the following cleanup code runs:

Ran cleanup code inside using host block.
Ran most service's cleanup code.

X:\path\to\App.exe (process nnn) exited with code 456.

Note that the return code for a normal ProcessExit case (rather than one simulated by Environment.Exit) would likely be 3221225786 (STATUS_CONTROL_C_EXIT) on Windows for v2.2.0 of this package and 0 for the latest (I think; based on ConsoleLifetime source).

Additional context

Trapping ProcessExit and then deciding how long to wait overall is rather problematic; it could wait too long/block, as in dotnet/extensions#1363, or too short, as here. The main options I can think of are:

1. Only have ProcessExit trigger the start of graceful shutdown (like Ctrl+C does), and give something else the responsibility to keep the shutdown from completing until Program.Main finishes.
2. Have some automatic way to know when Program.Main returns. Not sure if this is possible - I tried doing a thread.Join on the main thread, but that doesn't quite work. Polling it for when IsBackground flips to false looks like it could work, but that seems not exactly ideal.
3. Push the control up to the user for when to have the shutdown handler terminate (have the user pass in an event signaled from Program.Main maybe or something roughly like that; more complicated API to make that work).

Overall, the host itself doesn't own the entirety of Program.Main, so having it decide when to stop running is tricky.

I did find that at least the logger cleanup can likely be done better by changing the order of the disposables in the service container, including by adding a first constructor parameter to Host that gets registered first - currently, the IHostApplicationLifetime is after loggers in the list of things to dispose, and the logic to dispose in reverse order on the service provider/scope means loggers don't get to cleanup.

For application insights, for example, having the logger not get to have Dispose called means it can't call Flush, so logs currently get lost on shutdown.

The full list of things that currently get lost are:

1. Earlier registered service provider disposables, including logger providers (likely could be fixed with some registration order/constructor chaining/dependency order changes).
2. Code after the using host block.
3. Program.Main's exit code.

[davidmatson]

A couple other things that likely currently get lost: full disposing of custom service providers, full disposing of custom hosts.

I ended up working around this with a wrapper around the entry point for now. Here's what it looks like:

class Program
{
    static void Main()
    {
        ProcessLifetimeHost.Run(MainCore);
    }

    static void MainCore(IProcessLifetime processLifetime)
    {
        new HostBuilder()
            // Configure etc
            .UseProcessLifetime(processLifetime)
            .Build()
            .Run();
    }
}

When run with the original set of things hooked up above, all of the cleanup code runs (including code outside the using block and setting the exit code).

The IProcessLifetime interface looks like this:

/// <summary>Defines a process lifetime.</summary>
interface IProcessLifetime
{
    /// <summary>
    /// Gets or sets the action to invoke to initiate graceful shutdown of currently-running application, if any.
    /// </summary>
    Action StopApplication { get; set; }
}

If you're interested in going down this route further, let me know and I can provide the other details as well (UseProcessLifetime extension method and ProcessLifetimeHost.Run method).

[analogrelay]

We'll likely need some corefx support on this one to give us an API to properly handle SIGTERM.

The rough proposal we have right now is to have a new API in corefx to hook the SIGTERM directly (right now we hook ProcessExit which can happen gracefully or via SIGTERM). In the new API, there would be a Cancel argument that can be used to stop the default behavior of the CLR (to shut down). In our lifetime we can hook that new event, stop the termination and signal the IApplicationLifetime to stop.

It means that if the app doesn't gracefully shutdown, SIGTERM won't shut it down, but SIGTERM is intended to be "graceful" so that makes sense. If a user needs to forcibly terminate the app, they can use SIGKILL

[davidmatson]

Unfortunately, I don't think it's possible for corefx to provide a cancellable API on Windows - the underlying signal sent by docker stop on Windows containers changed to CTRL_SHUTDOWN_EVENT, which can't be canceled. The MSDN docs could be a little clearer - they more say what can be done in this case (inline cleanup in the handler) rather than what can't be done (canceling the signal). See also a related moby issue for details.

[analogrelay]

Good to know. I was referring to graceful termination on Linux which sends SIGTERM but it may be more challenging on Windows.

This will be a tricky one, we'll have to investigate some options.

[MikhaelSorkin]

Hello!
Any update here? Is it possible to achieve graceful shutdown in windows container?

[davidmatson]

It is possible to achieve graceful shutdown in a Windows container, but it currently requires a bit of custom code to make it work. Specifically, it requires using P/Invoke to get a shutdown notification (see dotnet/extensions#2029 and moby/moby#25982). And then this signal cannot be canceled, so it requires different wiring to make it work with IHostBuilder/Host. The ProcessLifetimeHost snippet above shows how I'm currently doing that from a design standpoint (the implementation is rather complicated).

So yes it's possible, but it requires custom code and is not easy.

[MikhaelSorkin]

@davidmatson
Thank you for answer. Do you plan to publish your implementation? As a nuget package or as a source code?

[davidmatson]

My hope with this bug was to get a solution into the official product, so I wasn't planning on publishing my implementation. I'd have to check, but I could probably paste it here as a temporary workaround if there's interest.

[MikhaelSorkin]

@davidmatson
Well, i'm very interested in this workaround :)

[davidmatson]

Here's a zip of the source I'm using as a workaround. The usage is as described above. I'm using it with v2.2.0 of the hosting extensions package (it may work with a later version but I haven't tested that).

ProcessLifetimeWorkaround.zip

Disclaimer: as-is; just a workaround; not an official solution; not offering support for this code; etc.

[Tragetaschen]

This has cost me the better part of the day :-(

Whenever I ran my application on the command line on Linux, CTRL+C would gracefully exit and dispose my Autofac container. When running as systemd service, that would just start the disposing work, but then stop in the middle of it. Since that service would normally only be stopped when my embedded instrument with a volatile system log would be shut down, I had to unwrap a lot of layers before this became clear.

For the time being as a workaround, I have internalized the code for UseSystemd, replaced the ProcessExit with Console.CancelKeyPress and added KillSignal=SIGINT to my systemd unit.

Is there a planned path forward already? I think this should be more than "Backlog"…

[analogrelay]

Triage: The right approach here is twofold:

1. Get an event that runs before ProcessExit to capture SIGTERM (either by a full signal handler API or a SIGTERM) and gives the option to cancel the default behavior (terminating the process)

2. Change ConsoleLifetime to hook that event and use it to signal a graceful shutdown.

[analogrelay]

The above approach would also fix dotnet/extensions#1363

[davidmatson]

@anurse - the equivalent of SIGTERM on Windows/Docker (CTRL_SHUTDOWN_EVENT) can't be canceled. Thoughts?

[tmds]

Sharing my view of dotnet/extensions#1363 (comment).

ConsoleLifetime shouldn't be used by default.
Though there should be an API to Run a Host from Main that includes it.

That avoids the issue because the Host no longer assumes it gets disposed on ProcessExit.
And, not all hosts should be tied to these events, like a host that runs shortly as part of a larger application.