K3s Multus + Whereabouts doesn't work

[koendelaat]

Environmental Info:
K3s Version:

k3s version v1.30.0+k3s1 (14549535)
go version go1.22.2

Node(s) CPU architecture, OS, and Version:
Linux ph-lws-4c38d51ffd02-9840 5.14.21-150500.55.19-default #1 SMP PREEMPT_DYNAMIC Tue Aug 8 22:15:01 UTC 2023 (9908c29) x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server (planning for multiple nodes)

Describe the bug:
Unable to use K3s with Multus + Whereabouts.
Error: error adding container to network <network-name> : config file not found

Problem is related to whereabouts IPAM, switching to host-local IPAM makes the error disappear (but not desired)

Steps To Reproduce:

• Installed K3s
• Installed Multus + Whereabouts according to docs
• Create NAD:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: rto
  namespace: default
spec:
  config: |-
    {
        "cniVersion": "0.3.1",
        "type": "bridge",
        "bridge": "br0",
        "ipam": {
            "type": "whereabouts",
            "range": "10.0.0.0/16",
            ],
            "log_file" : "/tmp/whereabouts.log",
            "log_level" : "debug"
        }
    }

• Create Deployment that uses NAD:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: samplepod
  template:
    metadata:
      labels:
        app: samplepod
      annotations:
        k8s.v1.cni.cncf.io/networks: rto@eth2
    spec:
      containers:
        - name: samplepod
          image: alpine
          command:
            - /bin/ash
            - '-c'
            - 'trap : TERM INT; sleep infinity & wait'
          securityContext:
            capabilities:
              drop:
                - ALL
            runAsUser: 1000
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault

Expected behavior:
Pod started with event: Add eth2 [10.0.x.x/16] from default/rto

Actual behavior:
Pod pending with error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "42ece46993378dd1192b28fa08f48090fd490486fac9e99c94829ea9c4f05e4d": plugin type="multus" failed (add): [default/sample-deployment-59cf9c969c-zkqjg/378fdef6-0b26-4d53-a2ff-39b6f164fbb6:rto]: error adding container to network "rto": config file not found

Additional context / logs:
Adding a symbolic link for Whereabouts seems to help as a workaround
ln -s /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d /etc/cni/net.d/whereabouts.d

See also k8snetworkplumbingwg/whereabouts#465 (comment)

[MaciejKucia]

While it would be great if components like CNI plugins could respect custom directories, the ability to install k3s using "standard" paths (aka kubeadm paths) instead of /var/lib/rancher would solve a lot of pain points. I am using k3s with kubevirt and ''expected directory" issue seems to be the biggest hurdle.

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[brandond]

You can either pass the correct paths in the NetworkAttachmentDefinition, or run ln -s /var/lib/rancher/k3s/agent/etc/cni /etc/ on every node in the cluster.

I'm going to update the documentation to address this.