Nvidia driver 560.35.03 cannot access gpu

[valandyr]

Description

firejail cannot access nvidia gpu.
driver is functional outside of sandbox.

Steps to Reproduce

firejail --noprofile --private nvidia-smi

Expected behavior

Show GPU information/status.

Actual behavior

Failed to initialize NVML: GPU access blocked by the operating system

Behavior without a profile

Failed to initialize NVML: GPU access blocked by the operating system

Additional context

Last working driver is version 555.58.02

Environment

Linux archlinux 6.11.3-arch1-1
Arch Linux

firejail version 0.9.72
Compile time support:
- always force nonewprivs support is disabled
- AppArmor support is enabled
- AppImage support is enabled
- chroot support is enabled
- D-BUS proxy support is enabled
- file transfer support is enabled
- firetunnel support is disabled
- IDS support is disabled
- networking support is enabled
- output logging is enabled
- overlayfs support is disabled
- private-home support is enabled
- private-cache and tmpfs as user enabled
- SELinux support is disabled
- user namespace support is enabled
- X11 sandboxing support is enabled

Checklist

• The issues is caused by firejail (i.e. running the program by path (e.g. /usr/bin/vlc) "fixes" it).
• I can reproduce the issue without custom modifications (e.g. globals.local).
• The program has a profile. (If not, request one in https://github.com/netblue30/firejail/issues/1139)
• The profile (and redirect profile if exists) hasn't already been fixed upstream.
• I have performed a short search for similar issues (to avoid opening a duplicate).
  • I'm aware of browser-allow-drm yes/browser-disable-u2f no in firejail.config to allow DRM/U2F in browsers.
• I used --profile=PROFILENAME to set the right profile. (Only relevant for AppImages)

Log

Output of LC_ALL=C firejail /path/to/program

LC_ALL=C firejail --noprofile nvidia-smi
Parent pid 11709, child pid 11710
Child process initialized in 8.91 ms
Failed to initialize NVML: GPU access blocked by the operating system

Parent is shutting down, bye...

Output of LC_ALL=C firejail --debug /path/to/program

LC_ALL=C firejail --noprofile --debug nvidia-smi
Building quoted command line: 'nvidia-smi' 
Command name #nvidia-smi#
DISPLAY=:1 parsed as 1
Using the local network stack
Parent pid 11777, child pid 11778
Initializing child process
Host network configured
PID namespace installed
Mounting tmpfs on /run/firejail/mnt directory
Creating empty /run/firejail/mnt/seccomp directory
Creating empty /run/firejail/mnt/seccomp/seccomp.protocol file
Creating empty /run/firejail/mnt/seccomp/seccomp.postexec file
Creating empty /run/firejail/mnt/seccomp/seccomp.postexec32 file
Mounting /proc filesystem representing the PID namespace
Basic read-only filesystem:
Mounting read-only /etc
746 484 259:2 /etc /etc ro,relatime master:1 - ext4 /dev/nvme0n1p2 rw
mountid=746 fsname=/etc dir=/etc fstype=ext4
Mounting noexec /etc
747 746 259:2 /etc /etc ro,nosuid,nodev,noexec,relatime master:1 - ext4 /dev/nvme0n1p2 rw
mountid=747 fsname=/etc dir=/etc fstype=ext4
Mounting read-only /var
748 484 259:2 /var /var ro,relatime master:1 - ext4 /dev/nvme0n1p2 rw
mountid=748 fsname=/var dir=/var fstype=ext4
Mounting noexec /var
749 748 259:2 /var /var ro,nosuid,nodev,noexec,relatime master:1 - ext4 /dev/nvme0n1p2 rw
mountid=749 fsname=/var dir=/var fstype=ext4
Mounting read-only /usr
750 484 259:2 /usr /usr ro,relatime master:1 - ext4 /dev/nvme0n1p2 rw
mountid=750 fsname=/usr dir=/usr fstype=ext4
Mounting tmpfs on /var/lock
Mounting tmpfs on /var/tmp
Mounting tmpfs on /var/log
Create the new utmp file
Mount the new utmp file
Cleaning /home directory
Cleaning /run/user directory
Sanitizing /etc/passwd, UID_MIN 1000
Sanitizing /etc/group, GID_MIN 1000
Disable /run/firejail/sandbox
Disable /run/firejail/network
Disable /run/firejail/bandwidth
Disable /run/firejail/name
Disable /run/firejail/profile
Disable /run/firejail/x11
blacklist /run/firejail/dbus
Mounting read-only /proc/sys
Remounting /sys directory
Disable /sys/firmware
Disable /sys/hypervisor
Disable /sys/power
Disable /sys/kernel/debug
Disable /sys/kernel/vmcoreinfo
Disable /proc/sys/fs/binfmt_misc
Disable /proc/sys/kernel/core_pattern
Disable /proc/sys/kernel/modprobe
Disable /proc/sysrq-trigger
Disable /proc/sys/vm/panic_on_oom
Disable /proc/irq
Disable /proc/bus
Disable /proc/timer_list
Disable /proc/kcore
Disable /proc/kallsyms
Disable /usr/lib/modules/6.11.3-arch1-1/build (requested /usr/src/linux)
Disable /usr/lib/modules (requested /lib/modules)
Disable /boot
Disable /dev/port
Disable /run/user/1000/gnupg
Disable /run/user/1000/systemd
Disable /dev/kmsg
Disable /proc/kmsg
Disable /sys/fs
Disable /sys/module
Current directory: /home/constance
DISPLAY=:1 parsed as 1
Masking all X11 sockets except /tmp/.X11-unix/X1
Mounting read-only /run/firejail/mnt/seccomp
791 743 0:74 /seccomp /run/firejail/mnt/seccomp ro,nosuid - tmpfs tmpfs rw,mode=755,inode64
mountid=791 fsname=/seccomp dir=/run/firejail/mnt/seccomp fstype=tmpfs
Seccomp directory:
ls /run/firejail/mnt/seccomp
drwxr-xr-x root     root             120 .
drwxr-xr-x root     root             160 ..
-rw-r--r-- constance constance         640 seccomp
-rw-r--r-- constance constance         432 seccomp.32
-rw-r--r-- constance constance           0 seccomp.postexec
-rw-r--r-- constance constance           0 seccomp.postexec32
No active seccomp files
Drop privileges: pid 1, uid 1000, gid 1000, force_nogroups 0
Closing non-standard file descriptors
Starting application
LD_PRELOAD=(null)
execvp argument 0: nvidia-smi
Child process initialized in 13.21 ms
Searching $PATH for nvidia-smi
trying #/usr/local/sbin/nvidia-smi#
trying #/usr/local/bin/nvidia-smi#
trying #/usr/bin/nvidia-smi#
Failed to initialize NVML: GPU access blocked by the operating system
monitoring pid 2

Sandbox monitor: waitpid 2 retval 2 status 4352

Parent is shutting down, bye...

[pjhfggij]

UPDATE this suggestion solved this for me
not sure why arch being a rolling distro still hasn't picked up these changes and everyone needs to look this up on individual basis
add to corresponding profiles:

noblacklist /sys/module
whitelist /sys/module/nvidia*
read-only /sys/module/nvidia*

I've tried previously different combinations of these lines, but somehow they didn't do the trick for some reason - not sure why:

whitelist /sys/module/nvidia*
noblacklist /sys/module/nvidia*
ignore read-only /sys/module/nvidia*

I can reproduce this (Driver Version: 560.35.03, Arch)
also I first noticed issues with nvidia card (optmius laptop) in March this year, when trying to run wine under firejail, it wasn't able to access nvidia card, but it worked fine without firejail
I believe that it previously worked
Tried adding this to wine.local, but to no effect:

noblacklist /dev/nvidia*
noblacklist /usr/lib/nvidia*
noblacklist /usr/lib/x86_64-linux-gnu/libGL.so*
noblacklist /usr/lib/x86_64-linux-gnu/libvulkan.so*

[kmk3]

@pjhfggij on Oct 18:

UPDATE this
suggestion
solved this for me not sure why arch being a rolling distro still hasn't
picked up these changes and everyone needs to look this up on individual
basis add to corresponding profiles:

The changes from #6387 are only in the development version.

noblacklist /sys/module
whitelist /sys/module/nvidia*
read-only /sys/module/nvidia*

@valandyr

Does it work with the above lines?

Does it work with firejail-git (AUR)?

I've tried previously different combinations of these lines, but somehow they
didn't do the trick for some reason - not sure why:

whitelist /sys/module/nvidia*
noblacklist /sys/module/nvidia*
ignore read-only /sys/module/nvidia*

A noblacklist line should match an existing blacklist line.

See blacklist /sys/module in fs.c.

[rusty-snake]

Behavior without a profile

Try firejail --profile=noprofile /usr/bin/nvidia-smi.
If this fails, someone needs to strace into it.

[valandyr]

@rusty-snake
firejail --profile=noprofile /usr/bin/nvidia-smi works!
How does using a profile works and not using one fails?

In this context --noprofile --private=/path/to/dir --net=none --dbus-system=none --dbus-user=none --private-tmp --nonewprivs
If i replace --noprofile with --profile=noprofile can i expect the same behavior?

@kmk3
firejail --noprofile --whitelist=/sys/module/nvidia* --read-only=/sys/module/nvidia* /usr/bin/nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
Doesn't work.

[rusty-snake]

How does using a profile works and not using one fails?

--noprofile does not use a profile / an empty profile. noprofile.profile is a profile that does nothing beyond lifting (some) restrictions remaining with --noprofile (the most features is opt-in but a few are opt-out).

firejail --profile=noprofile /usr/bin/nvidia-smi works!

Play with the commands in noprofile.profile to findout which are required.

[valandyr]

(the most features is opt-in but a few are opt-out).

I assumed that --noprofile would disable everything, thanks for explaining.

Play with the commands in noprofile.profile to findout which are required.

firejail --noblacklist=/sys/module nvidia-smi works.

firejail --noblacklist=/sys/module/nvidia* nvidia-smi doesn't work.
Probably additional modules are necessary with driver version 560.

Feel free to close this issue whenever.

[rusty-snake]

firejail --noblacklist=/sys/module nvidia-smi works.
firejail --noblacklist=/sys/module/nvidia* nvidia-smi doesn't work.

As @kmk3 said: "A noblacklist line should [must] match an existing blacklist line."

Feel free to close this issue whenever.

"The changes from #6387 are only in the development version."

I leave it open for now (~1day). Anybody else, fell free to close.

---

Can anybody with firejail from git-master and a nvidia system check that firejail --profile=noprofile ls /sys/module actually outputs the same as ls /sys/module. If it outputs nothing (i.e. empty directory), we have a "bug" in noprofile.profile.

[kmk3]

Duplicate of #6372