`podman run` fails with Permission denied: OCI permission denied

[carbolymer]

/kind bug

Description

Podman crashes when trying to start container with --privileged.

Steps to reproduce the issue:

1. Running podman rootless. VirtualBox is installed on the same machine, which seems to interfere with /dev/ contents.

2. Run the following commands:

$ podman run --rm --privileged bash
Error: crun: error stat'ing file `/dev/vboxusb/001/002`: Permission denied: OCI permission denied
$ /usr/bin/ls -al /dev/vboxusb/001
total 0
drwxr-x--- 2 root vboxusers     80 May 18 08:14 .
drwxr-x--- 4 root vboxusers     80 May 18 08:14 ..
crw-rw---- 1 root vboxusers 189, 1 May 18 08:14 002
crw-rw---- 1 root vboxusers 189, 2 May 18 08:14 003
$ stat /dev/vboxusb/001/002
  File: /dev/vboxusb/001/002
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: 0,5	Inode: 749         Links: 1     Device type: 189,1
Access: (0660/crw-rw----)  Uid: (    0/    root)   Gid: (  108/vboxusers)
Access: 2022-05-18 08:14:36.426666413 +0200
Modify: 2022-05-18 08:14:36.426666413 +0200
Change: 2022-05-18 08:14:36.429999746 +0200
 Birth: -

Describe the results you received:
Podman fails to start the container with the error above

Describe the results you expected:
The privileged container is started.

Additional information you deem important (e.g. issue happens only occasionally):
n/a

Output of podman version:

podman version 4.1.0

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpuUtilization:
    idlePercent: 94.47
    systemPercent: 0.7
    userPercent: 4.83
  cpus: 24
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: monolake
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.17.7-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 903196672
  memTotal: 33605513216
  networkBackend: cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 17829457920
  swapTotal: 36029067264
  uptime: 7h 58m 22.24s (Approximately 0.29 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /home/carbolymer/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.8.2-1
      Version: |-
        fusermount3 version: 3.11.0
        fuse-overlayfs: version 1.8.2
        FUSE library version 3.11.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/carbolymer/.local/share/containers/storage
  graphRootAllocated: 1000400236544
  graphRootUsed: 713347825664
  graphStatus:
    Backing Filesystem: f2fs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 95
  runRoot: /run/user/1000/containers
  volumePath: /home/carbolymer/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651861110
  BuiltTime: Fri May  6 20:18:30 2022
  GitCommit: e4b03902052294d4f342a185bb54702ed5bed8b1
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): physical, VirtualBox installed aside.

[mheon]

--privileged has rootless Podman making a bind mount for every device node on the host to mount them into the container. It looks like we have permissions to determine that the mentioned device node exists, but not to stat it, which is probably necessary to bind-mount. Can you check was the permissions on the given node are?

[mheon]

Though, I'm very curious as to how Podman can determine the node exists, but crun can't stat it - that doesn't sound like it should be possible.

Potentially SELinux/Apparmor or some other security measure denying access?

[carbolymer]

No selinux / apparmor here. Affected user belongs to vboxusers group.

[Luap99]

I think podman still has access to this via the group but the oci runtime will drop all supplementary groups so it is unable to access it.
Try adding --group-add keep-groups to the run command. and see if this works.

[mheon]

Concur, that sounds reasonable.

[carbolymer]

I confirm. The command:

podman run --rm -it --privileged --group-add keep-groups bash

works.

[rhatdan]

BTW This only works with crun, not runc...

[dnso86]

Same issue came up here (podman version 4.3.1), and @Luap99 's solution fixed it - Thanks!
The image was this if anyone wants to reproduce it 🙂

[mheon]

Though, I'm very curious as to how Podman can determine the node exists, but crun can't stat it - that doesn't sound like it should be possible.

Potentially SELinux/Apparmor or some other security measure denying access?

[carbolymer]

No selinux / apparmor here. Affected user belongs to vboxusers group.

[Luap99]

I think podman still has access to this via the group but the oci runtime will drop all supplementary groups so it is unable to access it.
Try adding --group-add keep-groups to the run command. and see if this works.

[mheon]

Concur, that sounds reasonable.

[carbolymer]

I confirm. The command:

podman run --rm -it --privileged --group-add keep-groups bash

works.

[rhatdan]

BTW This only works with crun, not runc...

[dnso86]

Same issue came up here (podman version 4.3.1), and @Luap99 's solution fixed it - Thanks!
The image was this if anyone wants to reproduce it 🙂