Introduced in v3.1.15 - PublishSingleFile binary fails to start (cannot extract bundle) when run by systemd

[enclave-alistair]

Description

As of released runtime version 3.1.15, SDK v3.1.409 and SDK v3.1.115, a 'PublishSingleFile' binary fails to start when launched by systemd.

The AppHost can no longer extract the single file bundle:

May 27 12:24:48 fv-az72-422 systemd[1]: Started BundlingIssue.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failure processing application bundle.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failed to determine location for extracting embedded files
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: DOTNET_BUNDLE_EXTRACT_BASE_DIR is not set, and a read-write cache directory couldn't be created.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: A fatal error was encountered. Could not extract contents of the bundle

Interestingly, the binary runs just fine when launched as root using sudo, or run as a lower-privileged user, it's only when launched via systemd that the failure occurs.

This issue does not happen in 408 or prior 40* versions.

I've created a repo at https://github.com/enclave-alistair/netcore31-409-singlefile-systemd that has a minimal recreate for the problem, along with precise steps in a Github Action that verifies the published binary runs fine as root, but not under systemd.

I've defined a build matrix in Github Actions that executes 401 through 409, 114 and 115 and the run only fails on 409 + 115. See this latest result set for evidence: https://github.com/enclave-alistair/netcore31-409-singlefile-systemd/actions/runs/882234200.

Introduced by fix for CVE-2021-31204?

I find it curious that a CVE related to single-file applications (dotnet/announcements#185) was addressed in 3.1.15, and this is where the break occurs. I do wonder whether systemd has tmp folder management applied in such a way that causes bundle extraction to fail when the CVE fix is applied.

This issue is kind of a big deal for us to be honest, because we are now between a rock and a hard place of not wanting to ship code with a known vulnerability, but also needing to run under systemd on linux.

Configuration

Recreated on:

• SDK 3.1.409 / 3.1.115 - Runtime version 3.1.15
• Ubuntu 20.04
• x64
• I do not believe it is specific to the OS platform, but am not certain.

Regression?

This issue was introduced in the most recent release 3.1.15.

Other information

Systemd Unit

Unit File that fails is described below:

The path I use for the binary file does not make a difference.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

DOTNET_BUNDLE_EXTRACT_BASE_DIR Workaround

If I specify a DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable in the unit file, then this issue does not occur, but it shouldn't be required.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=/var/tmp/bundle-extract"
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

Tagging subscribers to this area: @agocke, @vitek-karas, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

As of released runtime version 3.1.15, SDK v3.1.409 and SDK v3.1.115, a 'PublishSingleFile' binary fails to start when launched by systemd.

The AppHost can no longer extract the single file bundle:

May 27 12:24:48 fv-az72-422 systemd[1]: Started BundlingIssue.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failure processing application bundle.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: Failed to determine location for extracting embedded files
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: DOTNET_BUNDLE_EXTRACT_BASE_DIR is not set, and a read-write cache directory couldn't be created.
May 27 12:24:48 fv-az72-422 BundlingIssue[1894]: A fatal error was encountered. Could not extract contents of the bundle

Interestingly, the binary runs just fine when launched as root using sudo, or run as a lower-privileged user, it's only when launched via systemd that the failure occurs.

This issue does not happen in 408 or prior 40* versions.

I've created a repo at https://github.com/enclave-alistair/netcore31-409-singlefile-systemd that has a minimal recreate for the problem, along with precise steps in a Github Action that verifies the published binary runs fine as root, but not under systemd.

I've defined a build matrix in Github Actions that executes 401 through 409, 114 and 115 and the run only fails on 409 + 115. See this latest result set for evidence: https://github.com/enclave-alistair/netcore31-409-singlefile-systemd/actions/runs/882234200.

Introduced by fix for CVE-2021-31204?

I find it curious that a CVE related to single-file applications (dotnet/announcements#185) was addressed in 3.1.15, and this is where the break occurs. I do wonder whether systemd has tmp folder management applied in such a way that causes bundle extraction to fail when the CVE fix is applied.

This issue is kind of a big deal for us to be honest, because we are now between a rock and a hard place of not wanting to ship code with a known vulnerability, but also needing to run under systemd on linux.

Configuration

Recreated on:

• SDK 3.1.409 / 3.1.115 - Runtime version 3.1.15
• Ubuntu 20.04
• x64
• I do not believe it is specific to the OS platform, but am not certain.

Regression?

This issue was introduced in the most recent release 3.1.15.

Other information

Systemd Unit

Unit File that fails is described below:

The path I use for the binary file does not make a difference.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

DOTNET_BUNDLE_EXTRACT_BASE_DIR Workaround

If I specify a DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable in the unit file, then this issue does not occur, but it shouldn't be required.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=/var/tmp/bundle-extract"
ExecStart=/usr/bin/mybinary

[Install]
WantedBy=multi-user.target

Author:	enclave-alistair
Assignees:	-
Labels:	area-Single-File, untriaged
Milestone:	-

[enclave-alistair]

This may be a regression of #3846?

[vitek-karas]

Starting with that release the default extraction location has changed to $HOME/.net (previously it was in /var/tmp/.net or /tmp/.net). As you mention, you can redirect by setting DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable.

[enclave-alistair]

So, I'm guessing the failure is because systemd does not set $HOME?

[enclave-alistair]

This improves the quality of the workaround we can apply. I can use the %h expansion supported by systemd for the running user's home directory, we can get a robust (rather than hard-coded) value for DOTNET_BUNDLE_EXTRACT_BASE_DIR.

[Unit]
Description=BundlingIssue
After=network.target

[Service]
Environment="COREHOST_TRACE=1"
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=%h/.net/bundling"
ExecStart=$PWD/bin/Release/netcoreapp3.1/linux-x64/publish/BundlingIssue

[Install]
WantedBy=multi-user.target

See the workaround branch in the recreation repo, combined with the successful run at https://github.com/enclave-alistair/netcore31-409-singlefile-systemd/actions/runs/882403193.

[VSadov]

Right. $HOME is a POSIX standard environment variable, but I guess systemd is special in this regard.

[jjxtra]

Issues still exists in latest .net 5 sdk, is there a plan to fix it for .net 6?

[agocke]

@jjxtra Can you set DOTNET_BUNDLE_EXTRACT_BASE_DIR as described above?

[agocke]

@VSadov Can you confirm that this is only a problem if extraction is necessary? If the bundle doesn't use extraction I wouldn't expect this to be a problem.

[jjxtra]

@jjxtra Can you set DOTNET_BUNDLE_EXTRACT_BASE_DIR as described above?

Yes, but it seems like this shouldn't be needed...

[agocke]

Doing otherwise appears to be a security vulnerability.

[vitek-karas]

@jjxtra What would be the correct location under systemd? The extraction location needs:

• Writable (and executable)
• Not accessible by other users on the system (definitely not for writing, reading might be OK)
• No way for other users to subvert the location by creating preexisting files/folders there (this leads to DoS since the extraction would fail to create the exraction dir anyway)

Unfortunately /tmp or /var/tmp does not fulfill these requirements, so we moved to $HOME instead. Since $HOME doesn't exist under systemd, what would be the other location? And even if there is one, I don't know if it's a good idea to hardcode knowledge of systemd's special behavior into the runtime.

In general there so many different behaviors across various Linux distros and environments that I personally would prefer to follow a standard ($HOME is defined in POSIX) and leave everything else up to the developer to tweak via the environment variable. If you think environment variable is not suitable, we can definitely discuss adding some other mechanism to modify the default extraction path.

[enclave-alistair]

Personally I agree it's not a good idea to embed knowledge of systemd's "specialness" into the runtime.

Since in order to run a service under systemd, you must define a .service file anyway, in which there are clear ways to define an Environment variable that does use the correct user's home directory (via %h) in the same way as $HOME, that workaround may be sufficient, e.g.:

[Service]
Environment="DOTNET_BUNDLE_EXTRACT_BASE_DIR=%h/.net"

I would propose two things:

• Can we improve the message displayed to the user by the runtime if $HOME is not defined? At the moment it just says "Failed to determine location", but if it said "No $HOME variable is available, so extract location could not be determined.", that might make it clearer what is going on.
• There should probably be explicit documentation somewhere that setting the environment variable with %h is the recommended workaround to use when running under systemd, to avoid users accidentally blundering into using /var/tmp or /tmp, which reintroduces the previous security flaw.