Linux systemd conflict - help requested

[RichardHaselgrove]

We recently resolved - #4105 - a problem where a systemd configuration entry (PrivateTmp=true) was found to be the culprit in making idle detection fail.

I'm now encountering a different problem, which appears to require that we set PrivateTmp=true again, to work round a different problem. Obviously these two needs are incompatible.

The new problem arises at GPUGrid, where they are beta-testing a new app which requires the Wrapper app to perform a software installation. The wrapper command is

    <task>
        <application>/usr/bin/flock</application>
	<command_line>$PROJECT_DIR/miniconda.lock -c "/bin/bash ./miniconda-installer.sh -b -u -p $PROJECT_DIR/miniconda &&
                      $PROJECT_DIR/miniconda/bin/conda install -m -y -p gpugridpy --file requirements.txt "
	</command_line>
	<setenv>PATH=/usr/bin:/bin</setenv>
	<time_limit>1800</time_limit>
	<weight>10</weight>
    </task>

If PrivateTmp=false (the new default), this command fails with

21:48:18 (21729): wrapper: running /usr/bin/flock (/var/lib/boinc-client/projects/www.gpugrid.net/miniconda.lock -c "/bin/bash ./miniconda-installer.sh -b -u -p /var/lib/boinc-client/projects/www.gpugrid.net/miniconda &&
                      /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/bin/conda install -m -y -p gpugridpy --file requirements.txt ")
[21755] INTERNAL ERROR: cannot create temporary directory!
[21759] INTERNAL ERROR: cannot create temporary directory!
21:48:19 (21729): /usr/bin/flock exited; CPU time 0.118700
21:48:19 (21729): app exit status: 0x1
21:48:19 (21729): called boinc_finish(195)

If PrivateTmp=true, the wrapper task succeeds, and the full BOINC job carries on to a successful conclusion.

Could the systemd crew - @BryanQuigley, @Germano0, @smoe - please put their thinking caps on and suggest how this might be resolved? I'm assuming that the basic problem in the 'false' case is that the system /tmp/ structure is read-only to BOINC?

https://man7.org/linux/man-pages/man5/systemd.exec.5.html has details which may be relevant:

   PrivateTmp=

       Takes a boolean argument. If true, sets up a new file system
       namespace for the executed processes and mounts private /tmp/ and
       /var/tmp/ directories inside it that are not shared by processes
       outside of the namespace. This is useful to secure access to
       temporary files of the process, but makes sharing between
       processes via /tmp/ or /var/tmp/ impossible. 

[BryanQuigley]

They should not be installing in such a manner IMHO - and I'm strongly against making any of these changes upstream but you can try:
either adding /var/tmp and /tmp to ReadWritePaths=
or commenting out:
ProtectSystem=strict

[RichardHaselgrove]

Thanks. The beta test community at GPUGrid are having success with reverting the PrivateTmp setting. I'll pass on your other suggestions, as well as trying them myself.

But it appears to me that the root of the problem lies in the Linux design itself. I can see the logic in using Xserver to detect whether (specifically) the mouse is idle - after all, you won't be using a mouse if you can't see the pointer on a GUI screen. But why bury system configuration information deep in the /tmp/ structure? That creates the problem of secure system services not having (read) access to public system information. It seems like the left hand didn't know what the right hand was doing when these two different design decisions were taken.

The original Xserver idle detection routine was introduced by 69abe83, driven by "man Xserver". The only part of that which failed (leading to #4105) was getting the list of server names from /tmp/. Is that the only way of getting the name-list? Really?

[Germano0]

Xorg should not be a source for detecting user idle time. It never worked properly with boinc running as service. Moreover we keep ignoring that Wayland is the new standard.
#1187 (comment)

[BryanQuigley]

As @Germano0 said Xorg is quite the old standard. Systemd has improved security options a bunch in ways that we're not considered when the Xserver was written.

Either way, the project has the BOINC writable directory - why not use that? You can make your own tmp directory there if I'm not mistaken (haven't tried it)

[aaronpuchert]

Either way, the project has the BOINC writable directory - why not use that? You can make your own tmp directory there if I'm not mistaken (haven't tried it)

You can, but it won't be managed (e.g. cleaned up) by systemd-tmpfiles.

[RichardHaselgrove]

I'm hopeful that #4118 will resolve the idle detection issue, allowing the PrivateTmp setting to be reverted. But I haven't been able to locate the CI build for testing (I'll try again after the re-working of the CI system), and neither of the requested reviewers (@AenBleidd, @davidpanderson) have reported on the PR yet.

But in general, when considering Linux design issues, please remember to assess both corporate settings with a dedicated tech support team, and private settings where the user/owner is their own tech support.

[aaronpuchert]

But in general, when considering Linux design issues, please remember to assess both corporate settings with a dedicated tech support team, and private settings where the user/owner is their own tech support.

Could you elaborate on this? I'd think that BOINC is mostly used in a private setting, since most companies wouldn't allow it. The settings are mostly coming from the distribution, and it's not necessarily wrong for a distribution to be safe by default, i.e. use PrivateTmp and other isolation techniques. These can of course (and perhaps should) be documented, including the usage constraints they imply.

[RichardHaselgrove]

I'm not sure what the balance is between private and corporate usage. I come from the 'private' wing, but people like @TheAspens have commented on, specifically, the use of World Community Grid internally within IBM.

But I'm not a native Linux-speaker. I can drive it, to the extent of building my own clients from source, but for complete installations I prefer to deploy from the distributions - and that is where the distinction between the private and corporate worlds becomes apparent.

For example, the repo maintainers, thinking corporately, assume that a non-specialist user will be denied the power to restart a system service locally (no root or sudo access). Therefore, they choose to remove the "Shut down connected client ..." menu action from their version of the BOINC Manager. That reasoning fails for the private user, because we do have the sudo/root password, and I'd prefer to have the full menus. I also recently reported a bug (#4100) where the distinction between 'Close Window' and 'Exit BOINC' becomes important - the repo maintainers also remove the second version of that pair, on the grounds that a full exit may (optionally) close down the client too, which they think their corporate users should be dissuaded from doing.

[aaronpuchert]

the use of World Community Grid internally within IBM.

Ok, that makes sense.

For example, the repo maintainers, thinking corporately, assume that a non-specialist user will be denied the power to restart a system service locally (no root or sudo access).

I guess I just have a problem with the word "corporate" here: some Linux distributions such as RedHat/Fedora and (open)SUSE are indeed developed by and for corporations, but many others aren't, and they have similar security settings.

Most distributions have security profiles for both single user and multi-user settings, but both allow starting/stopping a service only for root.

Therefore, they choose to remove the "Shut down connected client ..." menu action from their version of the BOINC Manager. That reasoning fails for the private user, because we do have the sudo/root password, and I'd prefer to have the full menus.

I agree that removing the actions at compile-time is wrong.

But having the root password is generally not enough, because I don't think the protocol tells you anything about how the process is running. You'd have to guess that the system is running systemd and how the service is named. Granted, a service file comes with BOINC, but it is until now considered optional. And while a graphical version of sudo exists, I'm not sure how widely it is used or whether it works with Wayland. (Wayland doesn't want graphical applications running as root, if I remember correctly.)

Users running a systemd-managed system should not have a problem starting, stopping, or restarting services, if not via systemctl then for example via GUI tools that the distribution provides like (open)SUSE's YaST.

I consider myself a private user, but I wouldn't want the manager to shutdown my client service (especially not implicitly on exiting the manager), precisely because I'm using it as a service to have it always running, regardless of whether I'm logged in or have the manager open.

If anything, the client should know whether it is set up as service (and possibly tell the manager), so that it can refuse RPC shutdown requests and only allow stopping via systemd in that case.