On Podman, unable to access local ports forwarded to container applications listening on the loopback interface (e.g. debug endpoints)

[rm3l]

/kind bug
/area dev
/area odo-on-podman

What versions of software are you using?

Operating System:
Fedora 37, kernel 6.1.6-200.fc37.x86_64

Output of odo version:
odo v3.5.0 (8dbf42e)

How did you run odo exactly?

$ mkdir /tmp/nodejs && cd /tmp/nodejs
$ odo init --name debug-nodejs --devfile nodejs --starter nodejs-starter
$ ODO_EXPERIMENTAL_MODE=t odo dev --platform=podman --debug

...
 •  Executing the application (command: debug)  ...
 -  Forwarding from 127.0.0.1:40001 -> 3000
 -  Forwarding from 127.0.0.1:40002 -> 5858

...

As expected, trying to access the forwarded application port (40001 here) works correctly:

$ curl -i http://localhost:40001
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 39
ETag: W/"27-Jw3xdDamjf5jthPp3a/zd95HG0Y"
Date: Wed, 18 Jan 2023 15:04:06 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Hello from Node.js Starter Application!%

But trying to access the application on the forwarded debug port (40002 here) returns an error:

$ curl -i http://localhost:40002
curl: (56) Recv failure: Connection reset by peer

Actual behavior

Accessing the application on the forwarded debug port (40002 here) returns an error:

$ curl -i http://localhost:40002
curl: (56) Recv failure: Connection reset by peer

Expected behavior

When running odo dev against a cluster, we get the expected HTTP response from the forwarded application debug port:

$ curl -i http://localhost:40002
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8

WebSockets request was expected

I was expecting the same behavior on Podman.

Any logs, error output, etc?

podman kube generate

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.3.1
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.ContainerType/debug-nodejs-app-runtime: container
    io.kubernetes.cri-o.SandboxID/debug-nodejs-app-runtime: 269e89e69b091b501843a3c29f0578568cd91f76496c92da296c57cef27c62c
    io.podman.annotations.autoremove/debug-nodejs-app-runtime: "FALSE"
    io.podman.annotations.init/debug-nodejs-app-runtime: "FALSE"
    io.podman.annotations.privileged/debug-nodejs-app-runtime: "FALSE"
    io.podman.annotations.publish-all/debug-nodejs-app-runtime: "FALSE"
  creationTimestamp: "2023-01-18T15:17:30Z"
  labels:
    app: debug-nodejs-app
  name: debug-nodejs-app
spec:
  automountServiceAccountToken: false
  containers:
  - args:
    - tail
    - -f
    - /dev/null
    env:
    - name: PROJECTS_ROOT
      value: /projects
    - name: DEBUG_PORT
      value: "5858"
    - name: PROJECT_SOURCE
      value: /projects
    image: registry.access.redhat.com/ubi8/nodejs-16:latest
    name: debug-nodejs-app-runtime
    ports:
    - containerPort: 3000
      hostPort: 40001
    - containerPort: 5858
      hostPort: 40002
    resources:
      limits:
        memory: 1Gi
    securityContext:
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
    volumeMounts:
    - mountPath: /projects
      name: odo-projects-debug-nodejs-app-pvc
    - mountPath: /opt/odo/
      name: odo-shared-data-debug-nodejs-app-pvc
  enableServiceLinks: false
  hostname: debug-nodejs-app
  restartPolicy: Always
  volumes:
  - name: odo-projects-debug-nodejs-app-pvc
    persistentVolumeClaim:
      claimName: odo-projects-debug-nodejs-app
  - name: odo-shared-data-debug-nodejs-app-pvc
    persistentVolumeClaim:
      claimName: odo-shared-data-debug-nodejs-app
status: {}

[rm3l]

🤔 Same behavior with the java-springboot and java-quarkus Devfiles.

[rm3l]

FWIW, I just discovered a podman port command and noticed we had several port mappings for each port. Not sure if it explains the issue, that said:

☸ kind-local-k8s-cluster in ~/w/t/6510-unable-to-access-the-forwarded-debug-port-of-the-node.js-starter-project-on-podman on  main [!] via ☕ v17.0.5 on ☁️   
$ podman port -a                        
aede5e4daea4    8080/tcp -> 0.0.0.0:40001
aede5e4daea4    5858/tcp -> 0.0.0.0:40002
bcafb172ed0b    8080/tcp -> 0.0.0.0:40001
bcafb172ed0b    5858/tcp -> 0.0.0.0:40002

[rm3l]

As noticed by @feloy, it works if the application listens on all interfaces (0.0.0.0), which is not the case for most apps running in debug mode (by default node --inspect=$DEBUG_PORT will listen on the loopback interface on 127.0.0.1:$DEBUG_PORT). And Podman does not seem to forward host ports to such container ports.
This behavior might also happen if the main application listens on localhost.

TODO (Scope of this issue after team discussion on 2023-01-23):

• Research if there are any other ways to expose ports on Podman (other than using hostPorts) that would work with container apps listening on localhost
• If not, document this behavior as a difference between K8s and Podman. Add this to the FAQ