pod networking fails if net.ipv4.ip_forward=0 running on rootless podman

[haodeon]

What Happened?

I am doing some testing with kind and minikube. I have run into a k8s networking issue which I am hoping someone with more experience can help explain.

I am running rootless podman on Pop_OS! which is based off Ubuntu. When I create a cluster using minikube, pod networking is not working. I initially thought this was a DNS issue but then when I looked at the coredns pod logs it tells me that it cannot reach 192.168.49.1:53: i/o timeout.

After further debugging I discover that by enabling net.ipv4.ip_forward it all works after recreating the cluster.

What is confusing to me is when I use kind to create a cluster it works just fine without net.ipv4.ip_forward being enabled.

From what I can tell minikube and kind are both using kindnet by default. I tried the bridge cni in minikube and it wouldn't work regardless of IP forwarding being enabled or disabled.

Is it intended to require net.ipv4.ip_forward to be enabled for minikube with rootless podman to work?

How is kind able to work with net.ipv4.ip_forward disabled?

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Podman

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[haodeon]

Haven’t found an answer yet but issue #17107 appears to be a duplicate of this one.

Have tested @juparog’s workaround by adding another name server, like 8.8.8.8 to /etc/resolv.conf. After minikube stop + start, CoreDNS is able to resolve lookups.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[haodeon]

/remove-lifecycle rotten

[SanjayVas]

It looks like this may affect rootless Docker as well. See #18667 (comment)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[haodeon]

/remove-lifecycle stale