podman run failed with pasta error with podman 5

[escape0707]

Issue Description

I'm not very experienced with how to troubleshoot this pasta error. But since the Podman 5 update, whenever I run a container with Podman, it exits with a pasta error.

I'm using Arch Linux, configured the network with systemd-networkd and systemd-resolved.

Steps to reproduce the issue

1. Update to podman 5
2. $ podman run --rm fedora:latest

Describe the results you received

Error: pasta failed with exit code 1:
Couldn't set IPv6 route(s) in guest: No such device

Describe the results you expected

The containers run.

podman info output

$ podman version
Client:       Podman Engine
Version:      5.0.0
API Version:  5.0.0
Go Version:   go1.22.1
Git Commit:   e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0-dirty
Built:        Fri Mar 22 07:30:17 2024
OS/Arch:      linux/amd64

$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 99.96
    systemPercent: 0.01
    userPercent: 0.03
  cpus: 64
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: gpu1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1041
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1041
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.1-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 122659880960
  memTotal: 134919507968
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1041/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_03_26.4988e2b-1
    version: |
      pasta 2024_03_26.4988e2b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1041/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 107935379456
  swapTotal: 107935379456
  uptime: 20h 3m 28.00s (Approximately 0.83 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/myuser/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/myuser/.local/share/containers/storage
  graphRootAllocated: 3936562970624
  graphRootUsed: 3369009352704
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1041/containers
  transientStore: false
  volumePath: /home/myuser/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.0
  Built: 1711060217
  BuiltTime: Fri Mar 22 07:30:17 2024
  GitCommit: e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0-dirty
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0

$ pacman -Qii podman
Name            : podman
Version         : 5.0.0-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache-2.0
Groups          : None
Provides        : None
Depends On      : catatonit  conmon  containers-common  crun  gcc-libs  glibc  iptables  device-mapper  libdevmapper.so=1.02-64  gpgme  libgpgme.so=11-64  libseccomp
                  libseccomp.so=2-64  passt
Optional Deps   : apparmor: for AppArmor support
                  btrfs-progs: support btrfs backend devices
                  cni-plugins: for an alternative container-network-stack implementation
                  fuse-overlayfs: for storage driver in rootless environment [installed]
                  slirp4netns: for alternative rootless network support [installed]
                  podman-compose: for docker-compose compatibility [installed]
                  podman-docker: for Docker-compatible CLI [installed]
Required By     : podman-compose  podman-docker
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 76.85 MiB
Packager        : David Runge <dvzrv@archlinux.org>
Build Date      : Fri 22 Mar 2024 07:30:17 AM JST
Install Date    : Thu 28 Mar 2024 08:44:55 AM JST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature
Backup Files    : None
Extended Data   : pkgtype=split

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

If revert to podman 4.9.3, the container runs without problems. I can use it with Nvidia GPUs.

[sbrivio-rh]

@escape0707, thanks for reporting this. It looks like pasta is failing to configure IPv6 connectivity for the container here. Can you please share what interfaces (ip link show), addresses (ip address show) and IPv6 routes (ip -6 route show) you have configured on the host?

By the way, as a temporary workaround, you can probably start your containers with IPv4-only connectivity, using podman run --net=pasta:-4 --rm fedora:latest.

[escape0707]

Thanks for the response. The --net=pasta:-4 trick did workaround this issue.

As for the network related info:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: enp67s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ae brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
    link/ether 50:e0:85:f3:30:95 brd ff:ff:ff:ff:ff:ff
5: enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
3: enp67s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b4:2e:99:a6:cf:ae brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 50:e0:85:f3:30:95 brd ff:ff:ff:ff:ff:ff
5: enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fea6:cfad/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
    inet 203.178.135.37/25 brd 203.178.135.127 scope global enp67s0f1.222
       valid_lft forever preferred_lft forever
    inet6 2001:200:0:1cd1:b62e:99ff:fea6:cfad/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591922sec preferred_lft 604722sec
    inet6 2001:200:0:1cd1::37/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b62e:99ff:fea6:cfad/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

$ ip -6 route
2001:200:0:1cd1::/64 dev enp67s0f1.222 proto kernel metric 256 pref medium
2001:200:0:1cd1::/64 dev enp67s0f1.222 proto ra metric 1024 expires 2591819sec hoplimit 64 pref medium
fe80::/64 dev enp67s0f1 proto kernel metric 256 pref medium
fe80::/64 dev enp67s0f1.222 proto kernel metric 256 pref medium
default proto static metric 1024 pref medium
        nexthop via 2001:200:0:1cd1::1 dev enp67s0f1.222 weight 1
        nexthop via fe80::7afe:3dff:fed4:7181 dev enp67s0f1.222 weight 1

[sbrivio-rh]

6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

This might explain it, the VLAN interface used for the default route is probably confusing pasta's netlink implementation. I should be able to reproduce it now, thanks, I'll keep you posted.

[hafu]

Running into the same issue without VLANs:

$ podman run quay.io/podman/hello:latest
Error: pasta failed with exit code 1:
Couldn't set IPv6 route(s) in guest: No such device

Having more than one route:

$ ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default proto ra metric 600 pref medium
        nexthop via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 weight 1
        nexthop via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3 weight 1

After deleting one of them:

# ip -6 ro de default via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3
# ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 proto ra metric 600 pref medium

It works:

$ podman run quay.io/podman/hello:latest
!... Hello Podman World ...!

Maybe it is confused by multiple default routes with same weight? Hope it helps.

[sbrivio-rh]

Maybe it is confused by multiple default routes with same weight? Hope it helps.

Thanks, yes, it helps in the sense that we get ENODEV from the kernel also without a VLAN device, when we have RTA_NEXTHOP attributes (multipath routes) without RTA_OIF, so I'm focusing on nexthop attributes now.

I can't just reproduce it with any multipath route with two nexthops with the same weight, I think we need to have an actual mismatch of the interface numbering between host and container (expected, as we don't copy all the interfaces).

When I reproduce it with something similar to #22192 (comment), it looks like this:

[pid   159] sendto(7, [{nlmsg_len=148, nlmsg_type=0x18 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|NLM_F_MULTI|NLM_F_ACK|0x400, nlmsg_seq=14, nlmsg_pid=0}, "\x0a\x00\x00\x00\xfe\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x08\x00\x06\x00\x00\x04\x00\x00\x3c\x00\x09\x00"...], 148, 0, NULL, 0) = 148
[pid   159] recvfrom(7, [{nlmsg_len=168, nlmsg_type=NLMSG_ERROR, nlmsg_flags=0, nlmsg_seq=14, nlmsg_pid=162}, {error=-ENODEV, msg=[{nlmsg_len=148, nlmsg_type=0x18 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|NLM_F_MULTI|NLM_F_ACK|0x400, nlmsg_seq=14, nlmsg_pid=0}, "\x0a\x00\x00\x00\xfe\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x08\x00\x06\x00\x00\x04\x00\x00\x3c\x00\x09\x00"...]}], 65536, 0, NULL, NULL) = 168

Unfortunately strace(1) can't decode this because we have some padding in the message (I think that's the reason), but I'm now fairly sure we need to go "into" RTA_NEXTHOP attributes and also adjust interface numbers there as we copy host routes.

[sbrivio-rh]

pasta patch now pending review (testing also appreciated) at https://archives.passt.top/passt-dev/20240404151244.3216213-1-sbrivio@redhat.com/

[hafu]

Thanks. I applied the patch and it seems to work.

$ podman run --rm quay.io/podman/hello:latest

!... Hello Podman World ...!

$ ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default proto ra metric 600 pref medium
        nexthop via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 weight 1
        nexthop via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3 weight 1

IPv6 connectivity from inside a container also works.

[sbrivio-rh]

This is now fixed in pasta's version 2024_04_05.954589b -- I guess we can close it, unless @escape0707 prefers to wait for an updated Arch Linux package to check the fix.

[escape0707]

@sbrivio-rh Sorry, I was kinda busy yesterday. Thank you for your hard work. I'll test the patch today.

[escape0707]

I can confirm the patch fixed my issue. Thank you for maintaining this amazing tool! Closing this issue.