failed to reexec: Permission denied

[distributed-clouds]

Issue Description

Hello everyone I have built the podman from source on my ubuntu 24.04 on ec2 instance, everything looks good I can pull the images I can run the container, I can build the images. but each time when i reboot the instance or want to run the podman for the first time I get this error
failed to reexec: Permission denied
but when I repeat the command again it works:

podman version

Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.22.6
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Thu Nov 21 16:21:39 2024
OS/Arch:      linux/amd64

any idea?
thanks for your help

Steps to reproduce the issue

Steps to reproduce the issue

1. run any podman commands for the first time

Describe the results you received

failed to reexec: Permission denied

Describe the results you expected

return normal output

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 3d774d57d0ba1857573df331617520688f543444'
  cpuUtilization:
    idlePercent: 83.97
    systemPercent: 6.89
    userPercent: 9.14
  cpus: 2
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: file
  freeLocks: 2048
  hostname: ip-100-68-9-90
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-1019-aws
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3503554560
  memTotal: 4023279616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: Unknown
    path: /usr/local/libexec/podman/netavark
    version: netavark 1.14.0-dev
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.2.0+dev
      commit: v1.2.0-69-gb7da1673
      spec: 1.2.0
      go: go1.22.4
  os: linux
  pasta:
    executable: /usr/local/bin/pasta
    package: Unknown
    version: |
      pasta 2024_11_21.238c69f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: false
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 44.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/ubuntu/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ubuntu/.local/share/containers/storage
  graphRootAllocated: 30083776512
  graphRootUsed: 6272868352
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ubuntu/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 1732206099
  BuiltTime: Thu Nov 21 16:21:39 2024
  GitCommit: 874bf2c301ecf0ba645f1bb45f81966cc755b7da
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mheon]

Probably related to the rootless pause process, which we both create and try to join on first run, then only join on subsequent runs. You can do a podman system migrate and if that causes the error again that confirms the cause.

[distributed-clouds]

Hi @mheon Thanks for your answer, even for running this command
podman system migrate I get the same error for the first time, and after the second time and rebooting the system, nothing has changed.
BTW, there is no container running on the system

[Luap99]

Do you apparmor running, if so can you see if it logs anything or try to disable it.
The error here is just from a simple execvp ("/proc/self/exe", argv); so that should not be blocked normally.

[distributed-clouds]

Hi @Luap99 Thanks for you message after disabling the apparmor and rebooting the system,
now I can not running the podman at all :D

cannot clone: Permission denied
Error: cannot re-exec process

[distributed-clouds]

another update, I just fully uninstalled apparmor from the system and now I don't get that error anymore on any podman commands
but another error is that when I run the following command:

podman run -d nginx

I get this error

Error: loading seccomp profile () failed: seccomp not enabled in this build

but I built the podman without any BUILDTAGS just like the documentation:

Build Tags
Otherwise, if you do not want to build Podman with seccomp or selinux support you can add BUILDTAGS="" when running make.

make BUILDTAGS=""
sudo make install

and from the podman info command

  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: false
    seccompProfilePath: ""
    selinuxEnabled: false

[Luap99]

There really isn't a reason to to disable seccomp. I think maybe you need to use --security-opt seccomp=unconfined if you don't want it.
But running without seccomp is not something we recommend, I would say we should fix the build docs. I don't even know why we allow compiling without seccomp support.

cannot clone: Permission denied

That means we cannot create the user namespace which almost certainly is blokced by the new apparmor restrictions on ubuntu or some other seccomp profile when run in a nested container for example.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[stoggi]

I ran into the same issue, and found that AppArmor was denying an execution of /proc/self/exec:

$ sudo dmesg -t
audit: type=1400 audit(1736291136.943:128): apparmor="DENIED" operation="exec" class="file" info="Failed name lookup - disconnected path" error=-13 profile="unprivileged_userns" name="/proc/self/exe" pid=41842 comm="podman" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

So I added the following to /etc/apparmor.d/local/podman:

owner /proc/self/exe ix,
audit deny /proc/self/exe rw,

The first line allows the current owner of podman to run /proc/self/exe only if it is also owned by the same user, and inherit the AppArmor profile. The second line denies read and write access to /proc/self/exe, might not be needed since AppArmor denies by default but feels nice to be explicit.

And then verified the AppArmor profile can be parsed correctly with:

sudo apparmor_parser -r /etc/apparmor.d/podman

And all works well 👍

[Luap99]

@stoggi Can you report this to the upstream apparmor folks that maintain the podman profile?
We do not maintain a podman profile here so that should be fixed over there as we must be able to exec /proc/self/exe in many parts of our code.

I close this as it is not an upstream issue

[stoggi]

After some more investigation, it seems that I spoke too soon with the AppArmor profile change suggested above. It turns out it didn't quite solve the issue.

The podman profile that is provided by the AppArmor project is supposed to add an exception for podman with a default allow profile via the unconfined flag (man apparmor):

# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

abi <abi/4.0>,
include <tunables/global>

profile podman /usr/bin/podman flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/podman>
}

Which is applied and works correctly when installing Podman via apt which installs into /usr/bin/podman. But when you compile from source and run sudo make install Podman installs into /usr/local/bin/podman. And so this AppArmor profile is not applied (because it only applies to /usr/bin/podman). AppArmor then denies the execution of /proc/self/exe in the newly created restricted user namespace:

podman/pkg/rootless/rootless_linux.c

Lines 1100 to 1101 in 1a00c92

	execvp ("/proc/self/exe", argv);
	fprintf (stderr, "failed to reexec: %m\n");

So on the one hand, Ubuntu has introduced restricted user namespaces in AppArmor by default. And AppArmor did add an exception for Podman, when installed to /usr/bin/podman which is the case via the Ubuntu podman package.

But on the other hand, building Podman from source installs into /usr/local/bin/podman, but now in order to support the latest version of Ubuntu a new AppArmor profile could be installed that specifically allows /usr/local/bin/podman.

If the AppArmor project were to update the profile they could change:

profile podman /usr/{bin,local/bin}/podman flags=(unconfined) {

Which would apply the podman profile from both /usr/bin/podman and /usr/local/bin/podman. I'll suggest this, linking to this post for context.

[ibejoeb]

For now, on Ubuntu 24.04, it seems sufficient to manually edit the podman profile /etc/apparmor.d/podman and change the path to /usr/local/bin/podman.