singularity exec --oci fails on node without /run/user/<uid> directory

[nathanweeks]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release? (no, but I'm sure it impacts the 4.2.1 --- see below)
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Only issues that can be replicated on the latest release, or development branch, of SingularityCE will be investigated and fixed. The open source project does not maintain long-term stable branches or fix bugs in prior versions. If you require LTS support then please see the Sylabs website.

Version of Singularity

What version of Singularity are you using? Run:

$ singularity --version

singularity-ce version 4.1.5-1.el8

Describe the bug

singularity exec --oci (or run/shell) fails on a node where a /run/user/ directory has not been created.

To Reproduce

Steps to reproduce the behavior:

$ ls -l /run/user
total 0
$ cat Containerfile 
FROM docker.io/library/ubuntu:24.04
RUN apt update
$ singularity build --oci test.oci-sif Containerfile 
...
INFO:    Build complete: test.oci-sif
$ singularity exec --oci test.oci-sif ls
INFO:    System configuration does not support cgroup management - starting container in current cgroup
2024-11-08T12:00:00.00000Z: create directory `/run/user/1000`: No such file or directory

Expected behavior

Output of ls command

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?

Rocky Linux 8.9

Installation Method

RPM from EPEL repo

Additional context

My current understanding: the /run/user/<uid> directory is usually created by systemd-logind, but the pam_systemd module that registers user sessions with systemd-logind is disabled on compute nodes where the pam_slurm_adopt module has been enabled to only allow SSH access to compute nodes on which the user has a running Slurm job (I don't have any statistics, but I would suspect this is used on the majority of Slurm clusters). The aforementioned pam_slurm_adopt documentation includes example Slurm prolog and epilog scripts that loginctl enable-linger and loginctl disable-linger, respectively; though I surmise these are not implemented on the majority of Slurm clusters (more discussion here: https://support.schedmd.com/show_bug.cgi?id=5920).

singularity-ce's OCI launcher assumes the existence of /run/user/<uid> (for the rootless use case) at least at

singularity/internal/pkg/runtime/launcher/oci/oci_linux.go

Line 64 in 788e8d4

return fmt.Sprintf("/run/user/%s/singularity-oci", u.Uid), nil

It would be helpful if this path could be configurable by the user and/or sysadmin to accommodate such hosts where /run/user/<uid> does not exist.

[dtrudg]

I'm not sure I'd really classify this as a bug in Singularity... the pam_systemd / systemd-logind flow is a pretty fundemental expectation on all mainstream Linux distributions, which heavily integrate systemd components into the base installation / configuration. Lack of /run/user/UID is likely to affect more and more software over time, and the existence of the prolog / epilog scripts in the Slurm documentation is really an acknowledgement that using pam_slurm_adopt breaks general expectations on modern distributions.

I wish that Slurm would address through changes to the pam_slurm_adopt module, or otherwise... but recognise this seems unlikely and it is an impedement to the use of OCI mode where the prolog / epilog setup has not been applied.

We should definitely mention the issue in the documentation, and point to the prolog / epilog script notes.

We can likely add a fall back to using a runtime state directory under $TMPDIR, but this would then have to be a local filesystem.

[dtrudg]

Hi @nathanweeks - are you able to try with the change that has been made in a PR at: #3402

I believe this should work for you... but I haven't tested it with a completely absent /run/user session directory ... only some changes which simulate that in other ways.

[nathanweeks]

Thanks! I'll try building & testing that branch.