KVM: Cilium CNI not working with buildroot 2021.02 systemd

[ilya-zuyev]

After #11688 was merged, we noticed integration test failures for Cilium CNI.
All test setups with KVM driver had the same error - https://storage.googleapis.com/minikube-builds/logs/11791/b8ae8cd/KVM_Linux.html#fail_TestNetworkPlugins%2fgroup%2fcilium%2fDNS

ilyaz@myhost --- g/minikube ‹buildroot-2021.02› » out/minikube start -p test --driver=kvm --cni=cilium                                                                                                                                            130 ↵
* [test] minikube v1.21.0 on Ubuntu 21.04
* Using the kvm2 driver based on user configuration
* Starting control plane node test in cluster test
* Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.20.7 on Docker 20.10.6 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Configuring Cilium (Container Networking Interface) ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass

! /home/ilyaz/google-cloud-sdk/bin/kubectl is version 1.17.17-dispatcher, which may have incompatibilites with Kubernetes 1.20.7.
  - Want kubectl v1.20.7? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "test" cluster and "default" namespace by default

ilyaz@myhost --- g/minikube ‹buildroot-2021.02› » kubectl  --context test  replace --force -f test/integration/testdata/netcat-deployment.yaml
deployment.apps/netcat replaced
service/netcat replaced

ilyaz@myhost --- g/minikube ‹buildroot-2021.02› » kubectl --context test exec -it deploy/netcat -- /bin/sh                                                                                                                                          1 ↵
/ # nslookup google.com
^C
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss


/ # uname -a
Linux netcat-66fbc655d5-phln5 4.19.194 #1 SMP Fri Jun 25 23:10:48 UTC 2021 x86_64 Linux


/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether da:a8:6b:5f:bf:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.244/32 scope global eth0
       valid_lft forever preferred_lft forever

ISO containing the issue: https://storage.googleapis.com/minikube-builds/iso/11688/minikube-v1.21.0-1624660371-11688.iso
Revert PR: #11799

[ilya-zuyev]

@afbjorklund it looks like 4.19.194 breaks cilium. One idea is that it could be related to BPF configuration

[medyagh]

I was looking for bpf in our source code I found we indirectly depend on it in go.sum

github.com/cilium/ebpf v0.0.0-20200110133405-4032b1d8aae3/go.mod h1:MA5e5Lr8slmEg9bt0VpxxWqJlO4iwu3FBdHUzV7wQVg=
github.com/cilium/ebpf v0.0.0-20200702112145-1c8d4c9ef775/go.mod h1:7cR51M8ViRLIdUjrmSXlK9pkrsDlLHbO8jiB8X8JnOc=
github.com/cilium/ebpf v0.2.0/go.mod h1:To2CFviqOWL/M0gIMsvSMlqe7em/l1ALkX1PyjrX2Qs=
github.com/cilium/ebpf v0.5.0/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJXRs=

[afbjorklund]

Unsurprisingly, this is yet another systemd bug: cilium/cilium#10645

Caused by the upgrade from systemd 240 to systemd 247, where this was introduced in 245 :

systemd/systemd@5d4fc0e

Need to add some patch, like UBUNTU-drop-kernel.-settings-from-sysctl-defaults-shipped.patch

---

It is unrelated to the kernel version, looks similar on both:

level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
level=info msg="|  _| | | | | |     |" subsys=daemon
level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
level=info msg="Cilium 1.8.0 f455c7e69 2020-06-22T16:14:29+02:00 go version go1.14.4 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: a8f292139e923b205525feb2c8a4377005904776/1.13.2/Modified/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (4.19.182) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=warning msg="CONFIG_LWTUNNEL_BPF optional kernel parameter is not in kernel configuration" subsys=probes
level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf

level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
level=info msg="|  _| | | | | |     |" subsys=daemon
level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
level=info msg="Cilium 1.8.0 f455c7e69 2020-06-22T16:14:29+02:00 go version go1.14.4 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: a8f292139e923b205525feb2c8a4377005904776/1.13.2/Modified/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (4.19.194) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=warning msg="CONFIG_LWTUNNEL_BPF optional kernel parameter is not in kernel configuration" subsys=probes
level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf

We could upgrade the kernel separately in the future, perhaps ?