hyperkube: kube-proxy 1.16.11 does not work on CentOS7

[dmitry-irtegov]

What happened:
We install k8s 1.16.11 on CentOS 7.7.1908 (Core) node using custom installer
kubelet and CNI binaries are extracted from k8s.gcr.io/hyperkube-amd64:v1.16.11 image.
Everything works, except kube-proxy (and, obviously, anything that depends on it).
It spams log with the message:

E0618 09:42:42.811741       7 proxier.go:1418] Failed to execute iptables-restore: exit status 4 (iptables-restore v1.8.2 (nf_tables): 
line 86: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-ARQWXEG2ZEM2P3WE
)

Service IP are not available from this node.

What you expected to happen:
kube-proxy works.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
This does NOT happen on 1.16.11/Ubuntu 18.04 and 1.16.10/Centos 7 on the same OS image.
Looks like iptables native/legacy mode detection is somehow broken on CentOS 7.

Environment:

• Kubernetes version (use kubectl version): v1.16.11
• Cloud provider or hardware configuration: vSphere, 4 CPU / 8192 MB RAM
• OS (e.g: cat /etc/os-release): CentOS Linux release 7.7.1908 (Core)
• Kernel (e.g. uname -a): 3.10.0-1062.9.1.el7.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP
• Install tools: custom
• Network plugin and version (if this is a network-related bug):
• Others:

[athenabot]

/sig network

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[justaugustus]

/sig release
/area release-eng
/priority critical-urgent
cc: @kubernetes/sig-network-bugs @kubernetes/release-engineering

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[superseb]

What binaries are you extracting? The base image for hyperkube got bumped from v1.16.10 to v1.16.11 (and 1.17.7 and 1.18.4) from Debian 9 (stretch) to Debian 10 (buster). And Debian 10 has uses nftables by default (https://wiki.debian.org/iptables). Still depends on what you are extracting from the hyperkube image if this is what is affecting you.

[dmitry-irtegov]

What binaries are you extracting?

/opt/cni/* and /hyperkube for k8s 1.16.*
/opt/cni/* and /usr/local/bin/kubelet for k8s 1.17 and 18

We do NOT extract iptables binary, and the error is obviously produced by kube-proxy container which we do not touch.

[aojea]

This seems a duplicate of #71305 (comment)

iptables-restore: exit status 4 (iptables-restore v1.8.2 (nf_tables):

[dmitry-irtegov]

This is not duplicate, because CentOS 7 does NOT use nf_tables, and that issue about machines which do (say, CentOS 8).

[aojea]

well, that's what the output you pasted says 😄 it seems it is using iptables-nft instead of iptables-legacy, that's what I assumed it was the same issue 😅

E0618 09:42:42.811741 7 proxier.go:1418] Failed to execute iptables-restore: exit status 4 (iptables-restore v1.8.2 (nf_tables):

[justaugustus]

/assign
I'm attempting a fix in #92354.

[justaugustus]

We've published v1.18.5-rc.1, v1.17.8-rc.1, and v1.16.12-rc.1, which include new hyperkube images.

Can you test these and let us know if this resolves your issue?
We're holding off on any patch releases until we get feedback here, so please let report back when you can.

cc: @kubernetes/release-engineering

[justaugustus]

k8s.gcr.io/hyperkube:v1.16.12 is live, as part of https://github.com/kubernetes/kubernetes/releases/tag/v1.16.12, so this should be resolved now.

Other patches:
https://github.com/kubernetes/kubernetes/releases/tag/v1.18.5 --> k8s.gcr.io/hyperkube:v1.18.5
https://github.com/kubernetes/kubernetes/releases/tag/v1.17.8 --> k8s.gcr.io/hyperkube:v1.17.8

/close

[k8s-ci-robot]

@justaugustus: Closing this issue.

In response to this:

k8s.gcr.io/hyperkube:v1.16.12 is live, as part of https://github.com/kubernetes/kubernetes/releases/tag/v1.16.12, so this should be resolved now.

Other patches:
https://github.com/kubernetes/kubernetes/releases/tag/v1.18.5 --> k8s.gcr.io/hyperkube:v1.18.5
https://github.com/kubernetes/kubernetes/releases/tag/v1.17.8 --> k8s.gcr.io/hyperkube:v1.17.8

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dmitry-irtegov]

Sorry, tested already on 1.16.12. Yes, it does solve the issue.

Thank you!!!