oci: buildkit TMPDIR fallback

[dtrudg]

Description of the Pull Request (PR):

Detect whether the ~/singularity.d/singularity-buildkit location can be used for buildkit. Full overlay support is required.

If not compatible, try falling back to a temporary directory with a warning that there will be no persistent cache.

If TMPDIR is not compatible, give a sensible fatal error.

Example behaviour

$ singularity build --oci test.sif Dockerfile
INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
WARNING: ~/.singularity filesystem does not support buildkit. Using temporary directory /tmp/singularity-buildkitd-3820174564. Layers will not be cached for future builds.
INFO:    singularity-buildkitd: running server on /run/user/1000/singularity-buildkitd/singularity-buildkitd-137340.sock
[+] Building 1.5s (4/6)                                                                                                                                                                
[+] Building 7.6s (6/6)                                                                                                                                                                
...
INFO:    Converting layers to SquashFS
INFO:    Cleaning up.
INFO:    Terminating singularity-buildkitd (PID 137371)
WARNING: removing singularity-buildkitd temporary directory /tmp/singularity-buildkitd-3820174564
INFO:    Build complete: test.sif

With an incompatible TMPDIR:

$ TMPDIR=/data/nfs-mnt singularity build --oci test.sif Dockerfile
INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
FATAL:   Temporary directory does not support buildkit. Please set $TMPDIR to a local filesystem.

This fixes or addresses the following GitHub issues:

• Fixes singularity build --oci fails when $HOME/.singularity/singularity-buildkitd is on NFS #3382

Before submitting a PR, make sure you have done the following:

• Read the Guidelines for Contributing, and this PR conforms to the stated requirements.
• Added changes to the CHANGELOG if necessary according to the Contribution Guidelines
• Added tests to validate this PR, linted with make check and tested this PR locally with a make test, and make testall if possible (see CONTRIBUTING.md).
• Based this PR against the appropriate branch according to the Contribution Guidelines
• Added myself as a contributor to the Contributors File

[nathanweeks]

(This is probably an separate issue) It appears that if XDG_RUNTIME_DIR is unset, the singularity-buildkitd unix domain socket is created in /run/singularity-buildkitd/ by default, which would fail in the rootless case (as on a multi-tenant HPC cluster):

$ singularity --debug build --oci test.oci-sif Dockerfile                                                              
DEBUG   [U=1000,P=3154663]persistentPreRun()            Singularity version: 3.7.3+3035-g82588d19a 
...
INFO    [U=0,P=3154678]    ensureBuildkitd()             Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
WARNING [U=0,P=3154678]    startBuildkitd()              ~/.singularity filesystem does not support buildkit. Using temporary directory /tmp/singularity-buildkitd-2111004501. Layers will not be cached for future builds.
DEBUG   [U=0,P=3154678]    startBuildkitd()              starting /home/user/local/singularity/libexec/singularity/bin/singularity-buildkitd [--root=/tmp/singularity-buildkitd-2111004501 --socket=unix:///run/singularity-buildkitd/singularity-buildkitd-3154678.sock]
FATAL:   singularity-buildkitd: permission denied
...

Setting XDG_RUNTIME_DIR to a user-writable path (which may be nonexistent before running singularity build) allows the build to succeed:

$ XDG_RUNTIME_DIR=/tmp/container-user-$UID  singularity build --oci test.oci-sif Dockerfile 
...
INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
WARNING: ~/.singularity filesystem does not support buildkit. Using temporary directory /tmp/singularity-buildkitd-4245275241. Layers will not be cached for future builds.
INFO:    singularity-buildkitd: running server on /tmp/container-user-1000/singularity-buildkitd/singularity-buildkitd-3155637.sock
...
WARNING: removing singularity-buildkitd temporary directory /tmp/singularity-buildkitd-4245275241
INFO:    Build complete: test.oci-sif

I may have had XDG_RUNTIME_DIR set in my environment when reporting #3382.

That said, this PR appears to address issue #3382 (including a helpful error when TMPDIR is set to a non-local file system).

[dtrudg]

(This is probably an separate issue) It appears that if XDG_RUNTIME_DIR is unset, the singularity-buildkitd unix domain socket is created in /run/singularity-buildkitd/ by default, which would fail in the rootless case (as on a multi-tenant HPC cluster):

$ singularity --debug build --oci test.oci-sif Dockerfile                                                              
DEBUG   [U=1000,P=3154663]persistentPreRun()            Singularity version: 3.7.3+3035-g82588d19a 
...
INFO    [U=0,P=3154678]    ensureBuildkitd()             Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
WARNING [U=0,P=3154678]    startBuildkitd()              ~/.singularity filesystem does not support buildkit. Using temporary directory /tmp/singularity-buildkitd-2111004501. Layers will not be cached for future builds.
DEBUG   [U=0,P=3154678]    startBuildkitd()              starting /home/user/local/singularity/libexec/singularity/bin/singularity-buildkitd [--root=/tmp/singularity-buildkitd-2111004501 --socket=unix:///run/singularity-buildkitd/singularity-buildkitd-3154678.sock]
FATAL:   singularity-buildkitd: permission denied
...

Setting XDG_RUNTIME_DIR to a user-writable path (which may be nonexistent before running singularity build) allows the build to succeed:

$ XDG_RUNTIME_DIR=/tmp/container-user-$UID  singularity build --oci test.oci-sif Dockerfile 
...
INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
WARNING: ~/.singularity filesystem does not support buildkit. Using temporary directory /tmp/singularity-buildkitd-4245275241. Layers will not be cached for future builds.
INFO:    singularity-buildkitd: running server on /tmp/container-user-1000/singularity-buildkitd/singularity-buildkitd-3155637.sock
...
WARNING: removing singularity-buildkitd temporary directory /tmp/singularity-buildkitd-4245275241
INFO:    Build complete: test.oci-sif

I may have had XDG_RUNTIME_DIR set in my environment when reporting #3382.

That said, this PR appears to address issue #3382 (including a helpful error when TMPDIR is set to a non-local file system).

Will tackle as a separate issue. Should be straightforward to follow a similar approache as #3402