Latest version of Podman available for RHEL 9 fails when trying to run container on disconnected host

[bfields3]

Issue Description

Using RHEL 9 hosts and the latest version of Podman available for RHEL 9, once a container image is downloaded/pulled, when trying to create/start the container in a disconnected environment, it fails with the error shown below:

Error: pasta failed with exit code 1:
External interface not usable

Steps to reproduce the issue

Steps to reproduce the issue
Each test used a container image pulled from quay.io. Each test performed the podman run command to start/run the container in a disconnected environment and connected as shown in attachment. Troubleshooting for Podman.pdf

1. On RHEL 9.5 vm on a Fedora 41 bare metal host using the latest version of podman for RHEL 9, check for the podman run command.
2. On a RHEL 9.4 vm on a Fedora 41 bare metal host using Podman version 4.9.4-rhel and updating Podman to version 5.2.2; run the podman run command.
3. On a bare-metal Fedora 41 host using the latest version of Podman (5.2.5), run the podman run command.

Describe the results you received

Please see attached for detailed test results:
Troubleshooting for Podman.pdf

Describe the results you expected

Expected results would include the successful output of the container being created/run after running the podman run command.

podman info output

Test 1: RHEL 9.5 host with Podman version 5.2.2
Result:
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: c0564282e9befb7804c3642230f8e94f1b2ba9f8'
  cpuUtilization:
    idlePercent: 63.87
    systemPercent: 13.58
    userPercent: 22.55
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.5"
  eventLogger: journald
  freeLocks: 2048
  hostname: rhel9.dtopvm
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-503.14.1.el9_5.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1756553216
  memTotal: 3836317696
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.2-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.16.1-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240806.gee36266-2.el9.x86_64
    version: |
      pasta 0^20240806.gee36266-2.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.el9.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4240437248
  swapTotal: 4240437248
  uptime: 0h 1m 31.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/befields/.config/containers/storRHEL 9.4 host with podman version 5.2.2age.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/befields/.local/share/containers/storage
  graphRootAllocated: 48301604864
  graphRootUsed: 14634754048
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/befields/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1729674539
  BuiltTime: Wed Oct 23 04:08:59 2024
  GitCommit: ""
  GoVersion: go1.22.7 (Red Hat 1.22.7-2.el9_5)
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Test 2 part 1: RHEL 9.4 host with podman version 4.9.4-rhel
Result:
host:
  arch: amd64
  buildahVersion: 1.33.11
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 92.71
    systemPercent: 2.43
    userPercent: 4.86
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: journald
  freeLocks: 2048
  hostname: redhat.vmdtop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-427.42.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1511460864
  memTotal: 3837399040
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240806.gee36266-2.el9.x86_64
    version: |
      pasta 0^20240806.gee36266-2.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4240437248
  swapTotal: 4240437248
  uptime: 0h 3m 20.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/befields/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/befields/.local/share/containers/storage
  graphRootAllocated: 48301604864
  graphRootUsed: 18659823616
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/befields/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1730450505
  BuiltTime: Fri Nov  1 03:41:45 2024
  GitCommit: ""
  GoVersion: go1.21.13 (Red Hat 1.21.13-5.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

Test 2 part 1: RHEL 9.4 host with podman version 5.2.2
Result:
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 89.15
    systemPercent: 4.05
    userPercent: 6.8
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: journald
  freeLocks: 2048
  hostname: redhat.vmdtop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-427.42.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1698029568
  memTotal: 3837399040
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240806.gee36266-2.el9.x86_64
    version: |
      pasta 0^20240806.gee36266-2.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4240437248
  swapTotal: 4240437248
  uptime: 0h 2m 39.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/befields/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/befields/.local/share/containers/storage
  graphRootAllocated: 48301604864
  graphRootUsed: 18660192256
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/befields/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1729674539
  BuiltTime: Wed Oct 23 04:08:59 2024
  GitCommit: ""
  GoVersion: go1.22.7 (Red Hat 1.22.7-2.el9_5)
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Test 3: Bare-metal Fedora 41 host with podman version 5.2.5
Result:
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 95.34
    systemPercent: 1.58
    userPercent: 3.08
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "41"
  eventLogger: journald
  freeLocks: 2048
  hostname: fedora-obsidian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.11.8-300.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7674245120
  memTotal: 67314667520
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0
  ociRuntime:
    name: crun
    package: crun-1.18.1-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.18.1
      commit: c41f034fdbb9742c395085fc98459c94ad1f9aae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241030.gee7d0b6-1.fc41.x86_64
    version: |
      pasta 0^20241030.gee7d0b6-1.fc41.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8550871040
  swapTotal: 8589930496
  uptime: 6h 3m 50.00s (Approximately 0.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/befields/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/befields/.local/share/containers/storage
  graphRootAllocated: 998500204544
  graphRootUsed: 163802521600
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/befields/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729209600
  BuiltTime: Thu Oct 17 19:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details:
Test 1 and 2 were RHEl 9 in a VM on a Fedora 41 bare-metal host. Test 3 was bare-metal Fedora 41 host.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

please define what a disconnected environment means ...

[bfields3]

Hello. Disconnected means no Internet connectivity.

…

On Tue, Nov 19, 2024, 3:08 PM Brent Baude ***@***.***> wrote: please define what a disconnected environment means ... — Reply to this email directly, view it on GitHub <#24614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOEEC2PLX6VW6YKPTGNVF6L2BOSEFAVCNFSM6AAAAABSDBRNPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBWG42TKOBVGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sbrivio-rh]

It depends on what you mean by "disconnected", as @baude pointed out.

If you have a network interface on the host that doesn't actually route traffic to the public internet, things should work (see also https://bugzilla.redhat.com/show_bug.cgi?id=2277954).

If you don't have a network interface at all, or the interface doesn't have addresses, pasta(1) needs (by default) to source addresses and routes from an interface to provide network connectivity to the container, so it will refuse to start.

At that point, the kind of network connectivity we should provide to the container is not clearly defined. It probably makes sense to require -net host in that case.

In any case, we're looking into options to improve usability on this topic, see also https://pad.passt.top/p/PastaWithoutNetworkConnectivity

[Jared-Sprague]

@sbrivio-rh Thank you for the reply! I want to second this bug as it is an issue for me too. My team develops an image that is designed to be run in air-gapped environments with podman. For the last few years running images with podman while not connected to the internet has been no issue. Now when we started getting the pasta error described by @bfields3.

Here is my most recent test, I just re-installed Fedora 41 on a bare-metal laptop and ran dnf update as of Nov 17th 2024 everything was up to date using podman version 5.2.5. Here are the results of my test:

with wifi connected to the internet, works fine:

clh@fedora:~$ podman run --rm -p 8080:8080 -p 8443:8443 --name mimir -d mimir:0.13.1
dc9a0659d61ad7c2e73de8e7bba5a79615713fca05ecb56c06a8b291e5d98807
clh@fedora:~$ 

With wifi network disconnected but interface enabled, pasta error:

clh@fedora:~$ podman run --rm -p 8080:8080 -p 8443:8443 --name mimir -d mimir:0.13.1
Error: pasta failed with exit code 1:
External interface not usable

with wifi disconnected from network and using --network host, works but with a warning:

clh@fedora:~$ podman run --network host --rm -p 8080:8080 -p 8443:8443 --name mimir -d mimir:0.13.1
Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
7a1cb7cf9818a033034f2fd3ff011e7766c1898ac51df940fa4483a25bee83b6

So @sbrivio-rh what appears to be happening is the case you described where if no network interface is available the pasta error is expected unless you add --network host.

However I certainly do have a network interface, and it works fine, In fact I'm submitting posting this comment using the same laptop from my tests above just with my wifi connected to the internet so I know for a fact that my network available, and enabled, and interface is working.

This looks like a bug in podman to me, the expected behavior should be the first case that you described "If you have a network interface on the host that doesn't actually route traffic to the public internet, things should work ", that expected behavior is not happening for me, as you can see from my tests above. Simply disconnecting from a wifi network causes the pasta error to occur unless I add --network host.

Podman should work fine without an internet connection without having to add a special option --network host, This is a problem because in the Public Sector many people using podman without an internet connection.

This should be very easy to reproduce as for anyone as well, on Fedora 41 with the latest updates, simply disconnect from wifi, or unplug your network cable and try running a container.

[sbrivio-rh]

However I certainly do have a network interface, and it works fine

Yes, but as far as I understand, it has no address and routes.

From my earlier comment (emphasis added in quote):

If you don't have a network interface at all, or the interface doesn't have addresses, pasta(1) needs (by default) to source addresses and routes from an interface to provide network connectivity to the container, so it will refuse to start.

This looks like a bug in podman to me

It's not an issue in Podman. It can be considered an issue in pasta (well, the behaviour is intended, but I see your problem). It's a separate component, just like, before Podman 5.x, you would use slirp4netns (also a different component) for rootless networking.

Podman should work fine without an internet connection without having to add a special option --network host, This is a problem because in the Public Sector many people using podman without an internet connection.

The problem is that pasta doesn't know which address it should use. By default, it copies one from the interface, but if there's no address, what should it do? You can also configure addresses and routes with pasta's options -a / --address and -g / --gateway. In slirp4netns, this was all hardcoded (and necessarily different from the host), but this also caused issues.

Until now, almost all the reports we got of "purposely" disconnected environments were from setups with an isolated network, but an interface with address and route, so we took care of that case. If this (different) case is relevant, then we should probably think of introducing some arbitrary default address and routes.

We didn't do it so far because pasta doesn't have a way to know that a network interface is currently missing an address, and if you bring up a container before the address is there, we don't want to silently continue with a behaviour that differs from the case where the interface already has an address assigned.

On the other hand, the non-determinism involved here was mostly problematic because of #22197, which has been solved now, so we might reasonably revisit it.

I would like to wait for some inputs from Podman developers before "fixing" this.

[sbrivio-rh]

@Luap99, long story short: we have at least two reports of users who would expect pasta to start even if they have a network interface that's not connected at all (no address, no route) and, at least in one case, it will never get connected.

This is different from cases with "isolated networks", intermittent connectivity, or quadlets at boot. What do you think? We could handle that in pasta (with a link-local address?) I suppose: now that #22197 is solved, the "timing at boot" part is not critical anymore.

[Luap99]

This is different from cases with "isolated networks", intermittent connectivity, or quadlets at boot. What do you think? We could handle that in pasta (with a link-local address?) I suppose: now that #22197 is solved, the "timing at boot" part is not critical anymore.

Do we need an address at all? If the host doesn't even have one there would be now way to connect to the host. (Well I guess the --map-host-loopback --map-gw would allow you to reach localhost so maybe we need one?)

I guess having a link local address inside should work fine if then also keeps working once the host interface gets a proper routable external address? I guess that would still need the monitoring part to update routes in the container? https://pad.passt.top/p/PastaWithoutNetworkConnectivity

Overall I am not sure if I would draw a difference between this case and the no connectivity early at boot that should be updated later (because well we can never know when connectivity will happen if at all).
We certainly fixed the early startup issue with quadlets but we need to do more #24637 and there are good chances that users wrote their own static units that suffer from the same thing.
So I think the monitor part is still relevant just no longer important for most.

---

@bfields3 If you have an issue in RHEL you should report that though the Red Hat support channels to ensure things get fixed in RHEL because anything we do upstream will only be fixed in a future version and then RHEL may or may not rebase to a new version but that would only happen with something like RHEL 9.6 never an older release. So to ensure fixes are backported when needed bugs must be reported through the Red Hat support channel. Linking upstream reports there is totally fine.

[sbrivio-rh]

This is different from cases with "isolated networks", intermittent connectivity, or quadlets at boot. What do you think? We could handle that in pasta (with a link-local address?) I suppose: now that #22197 is solved, the "timing at boot" part is not critical anymore.

Do we need an address at all? If the host doesn't even have one there would be now way to connect to the host. (Well I guess the --map-host-loopback --map-gw would allow you to reach localhost so maybe we need one?)

Right, we don't actually need an address to fix the two cases from this ticket. And for --map-host-loopback we would just need loopback addresses, which are going to be there anyway on lo.

The only problem with that is that we would force local connections to have loopback addresses (no --map-guest-addr), and that might push some users to adopt bad security practices as a result. Maybe the risk is reasonably small, though.

I guess having a link local address inside should work fine if then also keeps working once the host interface gets a proper routable external address?

Yes, at that point we would (implicitly) switch to NAT. Routes inside the container are not really that important, as long as they are usable and used.

I guess that would still need the monitoring part to update routes in the container? https://pad.passt.top/p/PastaWithoutNetworkConnectivity

Not necessarily: we could add that later on top (exactly because what route we set up in the container doesn't matter that much).

Overall I am not sure if I would draw a difference between this case and the no connectivity early at boot that should be updated later (because well we can never know when connectivity will happen if at all). We certainly fixed the early startup issue with quadlets but we need to do more #24637 and there are good chances that users wrote their own static units that suffer from the same thing. So I think the monitor part is still relevant just no longer important for most.

Right, I think so too. But we can have a fallback address even without that.

[Luap99]

Not necessarily: we could add that later on top (exactly because what route we set up in the container doesn't matter that much).

Right, I think so too. But we can have a fallback address even without that.

Yes you are right, if it defaults to NAT with a default route then there is no requirement to update the address later so I think this is the best solution.

[mwcz]

Just adding my 👍 to this. Last week I was doing some local development involving an image when my WiFi went out. Podman stopped working with the same error @bfields3 mentioned, and I was blocked from making further progress even though my work required no network connection (I work on an offline product). This scenario used to work, so thanks for your attention @sbrivio-rh & @Luap99. Looking forward to a patch!

[bfields3]

All, Thank you so much for your assistance on this. I also would like to mention that I did the same tests with Windows environment using a Windows 11 host and RHEL and Fedora VMs. Same results as what Jared got as well. Respectfully, Benny Fields III

…

On Thu, Nov 21, 2024 at 9:51 AM Michael Clayton ***@***.***> wrote: Just adding my 👍 to this. Last week I was doing some local development involving an image when my WiFi went out. Podman stopped working with the same error @bfields3 <https://github.com/bfields3> mentioned, and I was blocked from making further progress even though my work required no network connection (I work on an offline product). This scenario used to work, so thanks for your attention @sbrivio-rh <https://github.com/sbrivio-rh> & @Luap99 <https://github.com/Luap99>. Looking forward to a patch! — Reply to this email directly, view it on GitHub <#24614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOEEC2MGGUJHCYN6S5TZXJ32BX6RHAVCNFSM6AAAAABSDBRNPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJRGYYDIMRQGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Jared-Sprague]

Until now, almost all the reports we got of "purposely" disconnected environments were from setups with an isolated network, but an interface with address and route, so we took care of that case. If this (different) case is relevant, then we should probably think of introducing some arbitrary default address and routes.

@sbrivio-rh I think the case that is much more common will be cases like @mwcz described, where he is working on a laptop and his wifi goes down. There are a million reasons why this might happen and I'm sure all of us have experienced some of these:

Reasons why you might not be connected to a network:

1. You're on a plane
2. You're at a hotel and don't want to pay for the wifi
3. You're on the subway
4. You're you have a power outage at home
5. You're at a customer site and you're not allowed to connect to their network

I feel like in these cases podman run should still work fine without having to remember any special steps to avoid getting a "pasta" error.

All of us have container development workflows that we rely on, it will be very annoying if we have to alter these depending on if our network interface is connected to a network with an address or not, and have to swap back and forth depending on the state of our network.