Stack overflow in FileSystemWatcher when running as systemd service

[StephenCleary]

Description

.NET 9 app will fail to start when running as a systemd service; this appears to be a problem with FileSystemWatcher (for watching the json config files).

Reproduction Steps

Within WSL:

mkdir SystemdRepro && cd SystemdRepro
dotnet new worker
dotnet add package Microsoft.Extensions.Hosting.Systemd
dotnet build

Then create the file repro.service with the following content (you will need to adjust the User and ExecStart lines):

[Unit]
Description=repro
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=stephen
ExecStart=/mnt/d/Code/SystemdRepro/bin/Debug/net9.0/SystemdRepro

[Install]
WantedBy=multi-user.target

Then start the service and observe the logs:

sudo cp repro.service /etc/systemd/system && sudo systemctl daemon-reload
sudo systemctl start repro.service
journalctl -u repro.service -o cat | cat

Expected behavior

Service does not crash.

Actual behavior

Service crashes with these logs:

Started repro.service - repro.
Stack overflow.
Repeated 174575 times:
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.Write(System.Text.StringBuilder, Boolean)
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.GetPath(Boolean, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(System.String)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(System.String)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action`1<System.__Canon>, System.__Canon)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(Microsoft.Extensions.Configuration.FileConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider..ctor(Microsoft.Extensions.Configuration.Json.JsonConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(Microsoft.Extensions.Configuration.IConfigurationBuilder)
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationManager+ConfigurationSources.Add(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Microsoft.Extensions.Configuration.IConfigurationBuilder, System.Action`1<System.__Canon>)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, System.Action`1<Microsoft.Extensions.Configuration.Json.JsonConfigurationSource>)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, Microsoft.Extensions.FileProviders.IFileProvider, System.String, Boolean, Boolean)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, System.String, Boolean, Boolean)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.ApplyDefaultAppConfiguration(Microsoft.Extensions.Hosting.HostBuilderContext, Microsoft.Extensions.Configuration.IConfigurationBuilder, System.String[])
   at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(Microsoft.Extensions.Hosting.HostApplicationBuilderSettings)
   at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(System.String[])
   at Microsoft.Extensions.Hosting.Host.CreateApplicationBuilder(System.String[])
   at Program.<Main>$(System.String[])
repro.service: Main process exited, code=dumped, status=6/ABRT
repro.service: Failed with result 'core-dump'.

Regression?

No response

Known Workarounds

Disable reload-on-config-change:

var config = new ConfigurationManager();
config.AddInMemoryCollection(new Dictionary<string, string?>() { ["hostBuilder:reloadConfigOnChange"] = "false" });
var builder = Host.CreateApplicationBuilder(new HostApplicationBuilderSettings
{
	Args = args,
	Configuration = config,
});

Configuration

(within WSL):

$ dotnet --version
9.0.104
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble

Architecture: x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-extensions-configuration
See info in area-owners.md if you want to be subscribed.

[tarekgh]

@adamsitnik could you please have a look?

@dotnet/area-extensions-filesystem

[jeffhandley]

Thanks, @StephenCleary. We'll try to repro it with the steps you provided, and then we'll need to figure out if this is a regression from .NET 8 as that will guide the investigation. If you can try .NET 8 and share if it works for you there or not, please let us know.

[jeffhandley]

I confirmed the repro, and I also found this is not a regression. It fails the same way in .NET 8. The only thing I changed in the repro steps were the User and ExecStart as @StephenCleary indicated, and my ExecStart path was not a mounted drive from the host machine.

We will investigate from there.

[StephenCleary]

I confirm this is not a regression. I wasn't sure if the mounted drive would be part of the problem so thanks for that (I actually mount my dev drive, which I know the official docs say not to do but it seems to work...). Repro details below:

Changed my WSL setup to .NET 8:

• Uninstalled all .NET SDKs and runtimes (sudo apt remove --purge dotnet-runtime-* dotnet-sdk-*).
• Installed .NET 8.0 SDK (sudo apt-get update && sudo apt-get install -y dotnet-sdk-8.0).
• In SystemdRepro.csproj, changed TargetFramework from net9.0 to net8.0.
• In SystemdRepro.csproj, changed Microsoft.Extensions.Hosting version from 9.0.3 to 8.0.1.
• In SystemdRepro.csproj, changed Microsoft.Extensions.Hosting.Systemd version from 9.0.3 to 8.0.1.
• In repro.service, changed ExecPath from /mnt/d/Code/SystemdRepro/bin/Debug/net9.0/SystemdRepro to /mnt/d/Code/SystemdRepro/bin/Debug/net8.0/SystemdRepro.

dotnet --version now says 8.0.114.

After dotnet build, sudo cp repro.service /etc/systemd/system && sudo systemctl daemon-reload, and sudo systemctl start repro.service, then journalctl -u repro.service -o cat | cat shows the same error, just with a smaller number of recursive invocations before failure:

Started repro.service - repro.
Stack overflow.
Repeat 130834 times:
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.Write(System.Text.StringBuilder, Boolean)
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.GetPath(Boolean, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(System.String)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(System.String)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action`1<System.__Canon>, System.__Canon)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(Microsoft.Extensions.Configuration.FileConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider..ctor(Microsoft.Extensions.Configuration.Json.JsonConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(Microsoft.Extensions.Configuration.IConfigurationBuilder)
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationManager+ConfigurationSources.Add(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(Microsoft.Extensions.Configuration.IConfigurationSource)
   at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Microsoft.Extensions.Configuration.IConfigurationBuilder, System.Action`1<System.__Canon>)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, System.Action`1<Microsoft.Extensions.Configuration.Json.JsonConfigurationSource>)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, Microsoft.Extensions.FileProviders.IFileProvider, System.String, Boolean, Boolean)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(Microsoft.Extensions.Configuration.IConfigurationBuilder, System.String, Boolean, Boolean)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.ApplyDefaultAppConfiguration(Microsoft.Extensions.Hosting.HostBuilderContext, Microsoft.Extensions.Configuration.IConfigurationBuilder, System.String[])
   at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(Microsoft.Extensions.Hosting.HostApplicationBuilderSettings)
   at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(System.String[])
   at Microsoft.Extensions.Hosting.Host.CreateApplicationBuilder(System.String[])
   at Program.<Main>$(System.String[])
repro.service: Main process exited, code=dumped, status=6/ABRT
repro.service: Failed with result 'core-dump'.

So, this same problem exists in .NET 8.0.

[StephenCleary]

P.S. I don't have a real-world case for this; I have yet to meet anyone actually using .NET in a systemd service (generally Docker is just easier). I just found this when coding up a demo for a talk on .NET background services.

[jeffhandley]

Thank you, @StephenCleary. That context is super helpful and your detailed reproduction steps made this really easy to set up and confirm.

Given you don't have real-world impact on this, I'm going to mark it as help wanted [up-for-grabs] Good issue for external contributors and put it into the Future milestone.