failed (add): failed to create macvlan: cannot assign requested address

[dmfrey]

I'm not able to assign addresses to my pod.

Multus is deployed with FluxCD, here is the HelmRelease:
https://github.com/dmfrey/home-gitops/blob/main/kubernetes/homelab-k8s-001/apps/network/multus/app/helmrelease.yaml

And my networks are configured with the following NetworkAttachmentDefinition:
https://github.com/dmfrey/home-gitops/blob/main/kubernetes/homelab-k8s-001/apps/network/multus/config/networkattachment-iot.yaml

My Talos cluster exposes the eth0.50 vlan

92.168.30.31   network     LinkStatus   eth0              13        ether               38:f7:cd:c6:11:4a                                 up           true
192.168.30.31   network     LinkStatus   eth0.30           12        ether      vlan     38:f7:cd:c6:11:4a                                 up           true
192.168.30.31   network     LinkStatus   eth0.50           11        ether      vlan     38:f7:cd:c6:11:4a                                 up           true

I'm applying a pod annotation to the pod in the HelmRelease:

pod:
  annotations:
     k8s.v1.cni.cncf.io/networks: |
      [{
        "name": "iot-vlan",
        "namespace": "network",
        "ips": ["192.168.50.8/24"],
        "mac": "f1:fd:59:0b:bd:2b"
      }]

When the pod starts its fire the following events.

Events:                                                                                                                                                                                                                                                                                            
  Type     Reason                  Age   From               Message                                                                                                                                                                                                                                
  ----     ------                  ----  ----               -------                                                                                                                                                                                                                                
  Normal   Scheduled               5s    default-scheduler  Successfully assigned download/admin-c5999f66d-djg6t to dmf-amd-003                                                                                                                                                                    
  Normal   AddedInterface          5s    multus             Add eth0 [172.22.0.238/32] from cilium                                                                                                                                                                                                 
  Warning  FailedCreatePodSandBox  4s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a56b77988d8ac693fdbaad50a8b0d03e237a689d6df2b59678e00b713d207da4": plugin type="multus" name="multus-cni-network" failed (add): [do
wnload/admin-c5999f66d-djg6t/2ff097cc-9ee0-48b3-a0b2-4d60efb80ab4:iot-vlan]: error adding container to network "iot-vlan": plugin type="macvlan" failed (add): failed to create macvlan: cannot assign requested address                                                                           

Is there some way to get around this issue?

Environment:

• 5 Talos Linux kubernetes nodes
• 3 control plane nodes
• 2 worker nodes
• Ubiquiti UDM Special Edition
• .30 vlan homelab (where the nodes live)
• .50 vlan iot

If there is anything else I can provide, please let me know. Thanks for any assistance.

[dougbtv]

Hey Daniel!

Ummm, so... alright, another reference config, here's the example I use for testing setting a static ip/mac at run time (e.g. via annotation)

https://gist.github.com/dougbtv/a1f2b40bb3e57429ab0dcd8ad4d96c42

I think you might need to add tuning into the CNI chain, my example:

"plugins": [{
          "type": "bridge",
          "bridge": "cnitest0",
          "name": "testbridge",
          "ipam": {
            "type": "static"
          }
        }, {
          "capabilities": { "mac": true, "ips": true },
          "type": "tuning"
        }]

and then your config...

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: iot-vlan
spec:
  config: |-
    {
    "cniVersion": "0.3.1",
    "name": "iot-vlan",
    "plugins": [
        {
            "type": "macvlan",
            "master": "eth0.50",
            "mode": "bridge",
            "ipam": {
                "type": "static",
                "routes": [
                    {
                        "dst": "192.168.50.0/24",
                        "gw": "192.168.50.1"
                    }
                ]
            }
        },
        {
            "capabilities": {
                "mac": true,
                "ips": true
            },
            "type": "tuning"
        }
    ]
}

Kinda just wildly pasted in an run through a json lint. I think that might be the piece that's missing is the tuning plugin.

Worth a shot at least.

[dougbtv]

Cool homelab setup, like, it's rad that it looks like you're doing some kind of isolation to handle the IoT network or something? So like you isolate that traffic on the pod to the IoT segment or something? Just curious what you're doing if you don't mind sharing, mostly I think it's neat.

[dmfrey]

@dougbtv Thanks!

I mostly am following the k8s-at-home generated projects.

There are 5 nodes in my cluster: 3 control-plane and 2 workers.

My main use case is this. I want to expose Home Assistant running in my homelab vlan (30) to my iot vlan (50) so that it can discover the devices and services running over there. This used to work, but for some reason, the last couple of months, it stopped and I was getting that message that it couldn't assign addresses. I have a test pod I've been playing with, but just sits there in a pending state with the message I posted.

There are other services I'd like to expose to that vlan as well, like ESPHome, etc. as well

Is there anything specific you'd like to know about it?

[dmfrey]

I updated the network definition, but I'm still getting the same error:

Events:                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  Type     Reason                  Age   From               Message                                                                                                                                                                                                                                                                                                                                                                                               
  ----     ------                  ----  ----               -------                                                                                                                                                                                                                                                                                                                                                                                               
  Normal   Scheduled               10s   default-scheduler  Successfully assigned download/admin-c5999f66d-vlvxv to dmf-amd-001                                                                                                                                                                                                                                                                                                                                   
  Normal   AddedInterface          10s   multus             Add eth0 [172.22.2.245/32] from cilium                                                                                                                                                                                                                                                                                                                                                                
  Warning  FailedCreatePodSandBox  10s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ad7bd8435ba9a9631a360a4f73254375f055f86a2df69eb58d22ce59542f2fd3": plugin type="multus" name="multus-cni-network" failed (add): [download/admin-c5999f66d-vlvxv/9bd3eb7e-0713-4998-8102-1e6419018123:iot-vlan]: error adding container to network "iot-vlan": plugin type="macvlan" failed (add): 
failed to create macvlan: cannot assign requested address                                                                                                                                                                                                                                                                                                                                                                                                         

Is there something potentially with cilium or anything I need to consider on the router (Mine is a Unifi Dream Machine Special Edition)?

[dougbtv]

you can try making that macvlan by hand on your system using ip commands, to see what it is. but it's at the netlink layer, like netlink is saying you can't assign that ip address.

can you try a different range maybe? try: https://github.com/k8snetworkplumbingwg/whereabouts if you want to, too

[dmfrey]

@dougbtv can you provide a resource that instructs how to do that by hand? I'm just getting back to this and am unsure how to proceed. I assume I can do this just in a busybox container, right?

[dmfrey]

Does this log indicate that I have anything configured incorrectly?

multus-daemon 2025-02-25T14:04:57Z [debug] conflistDel: &{d189dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb /var/run/netns/cni-1b7f5ba8-66b9-c727-122b-b4b66f324437 eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE kube-system] [K8S_POD_NAME busybox-6f6c985b77-cvmtk] [K8S_POD_INFRA_CONTAINER_ID d189dd14eeb850fc
eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb[] [K8S_POD_UID cdb2ecf5-f300-4991-ae4c-4176704f765b]] map[] },                                                                                                                                                                                                                    
multus-daemon {                                                                                                                                                                                                                                                                                                                   
multus-daemon   "cniVersion": "0.3.1",                                                                                                                                                                                                                                                                                            
multus-daemon   "name": "cilium",                                                                                                                                                                                                                                                                                                 
multus-daemon   "plugins": [                                                                                                                                                                                                                                                                                                      
multus-daemon     {                                                                                                                                                                                                                                                                                                               
multus-daemon        "type": "cilium-cni",                                                                                                                                                                                                                                                                                        
multus-daemon        "enable-debug": false,                                                                                                                                                                                                                                                                                       
multus-daemon        "log-file": "/var/run/cilium/cilium-cni.log"                                                                                                                                                                                                                                                                 
multus-daemon     }                                                                                                                                                                                                                                                                                                               
multus-daemon   ]                                                                                                                                                                                                                                                                                                                 
multus-daemon }                                                                                                                                                                                                                                                                                                                   
multus-daemon }]                                                                                                                                                                                                                                                                                                                  
multus-daemon ] [{apps/v1 ReplicaSet busybox-6f6c985b77 e7426fa9-9c4e-4a6c-901f-f2bb1bdc336b 0xc0002e5ea7 0xc0002e5ea8}] [] []},Spec:PodSpec{Volumes:[]Volume{},Containers:[]Container{Container{Name:busybox,Image:docker.io/library/busybox:latest,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resourc
es:ResourceRequirements{Limits:ResourceList{memory: {{314572800 0} {<nil>} 300Mi BinarySI},},Requests:ResourceList{cpu: {{50 -3} {<nil>} 50m DecimalSI},memory: {{157286400 0} {<nil>} 150Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMes
sagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,},},RestartPolicy:Always,TerminationGracePeriodSec
onds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:dmf-intel-002,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,SupplementalGroup
s:[],FSGroup:*0,RunAsGroup:*0,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:*true,Tolerations:[]Toleration{Toleration{K
ey:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:
[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*false,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,HostUsers:nil,SchedulingGates:[]PodSchedulingGate{},ResourceClai
ms:[]PodResourceClaim{},},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:PodReadyToStartContainers,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:
00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [busybox],},PodCondition{Type:ContainersRe
ady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [busybox],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UT
C,Reason:,Message:,},},Message:,Reason:,HostIP:192.168.30.42,PodIP:,StartTime:2025-02-25 14:04:41 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:busybox,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerSt
ate{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:docker.io/library/busybox:latest,ImageID:,ContainerID:,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:Burstable,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuse
s:[]ContainerStatus{},Resize:,ResourceClaimStatuses:[]PodResourceClaimStatus{},HostIPs:[]HostIP{HostIP{IP:192.168.30.42,},},},}, &{d189dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb /var/run/netns/cni-1b7f5ba8-66b9-c727-122b-b4b66f324437 eth0 K8S_POD_NAME=busybox-6f6c985b77-cvmtk;K8S_POD_INFRA_CONTAINER_ID=
d189dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb;K8S_POD_UID=cdb2ecf5-f300-4991-ae4c-4176704f765b;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system   [123 34 98 105 110 68 105 114 34 58 34 47 111 112 116 47 99 110 105 47 98 105 110 34 44 34 99 104 114 111 111 116 68 105 114 34 58 34 47 104 111 115 116 114 111
111 116 34 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 53 45 99 105 108 105 117 109 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 11
0 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 99 111 110 102 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 108 111 103 84 111 83 116 100 101
 114 114 34 58 116 114 117 101 44 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 34 97 117 116 111 34 44 34 110 97 109 101 34 58 34 109 11
7 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 125[]}, &{{true} <nil> busybox-6f6c985b77-cvmtk kube-system d18
9dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb cdb2ecf5-f300-4991-ae4c-4176704f765b}, [0xc000a0e000 0xc000a0e1e0], 1, <nil>                                                                                                                                                                                        
multus-daemon 2025-02-25T14:04:57Z [debug] getIfname: &{{0.3.1 iot-vlan  map[] {} {[]  [] []} map[] <nil> []} {0.3.1 iot-vlan false [0xc000399040 0xc000399110]} network/iot-vlan  f1:fd:59:0b:bd:2b  [192.168.50.8/24] [] <nil> <nil> false false false true   [123 10 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34
48 46 51 46 49 34 44 10 32 32 34 110 97 109 101 34 58 32 34 105 111 116 45 118 108 97 110 34 44 10 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 100 101 98 117 103 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 109 97 99 118 108 97 11
0 34 44 10 32 32 32 32 32 32 34 109 97 115 116 101 114 34 58 32 34 101 116 104 48 46 53 48 34 44 10 32 32 32 32 32 32 34 109 111 100 101 34 58 32 34 98 114 105 100 103 101 34 44 10 32 32 32 32 32 32 34 105 112 97 109 34 58 32 123 10 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 115 116 97 116 105 99 34 44 10 32 
32 32 32 32 32 32 32 34 114 111 117 116 101 115 34 58 32 91 10 32 32 32 32 32 32 32 32 32 32 123 32 34 100 115 116 34 58 32 34 49 57 50 46 48 46 48 46 48 47 56 34 44 32 34 103 119 34 58 32 34 49 57 50 46 49 54 56 46 53 48 46 49 34 32 125 10 32 32 32 32 32 32 32 32 93 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 3
2 32 32 123 10 32 32 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 109 97 99 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 105 112 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 32 32 125 44 10 32 32 32 32 32 32 32 32 34 
116 121 112 101 34 58 32 34 116 117 110 105 110 103 34 10 32 32 32 32 125 10 32 32 93 10 125[]}, eth0, 1                                                                                                                                                                                                                          
multus-daemon 2025-02-25T14:04:57Z [debug] LoadCNIRuntimeConf: net1, <nil> &{{0.3.1 iot-vlan  map[] {} {[]  [] []} map[] <nil> []} {0.3.1 iot-vlan false [0xc000399040 0xc000399110]} network/iot-vlan  f1:fd:59:0b:bd:2b  [192.168.50.8/24] [] <nil> <nil> false false false true   [123 10 32 32 34 99 110 105 86 101 114 115 10
 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 34 110 97 109 101 34 58 32 34 105 111 116 45 118 108 97 110 34 44 10 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 100 101 98 117 103 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 10
9 97 99 118 108 97 110 34 44 10 32 32 32 32 32 32 34 109 97 115 116 101 114 34 58 32 34 101 116 104 48 46 53 48 34 44 10 32 32 32 32 32 32 34 109 111 100 101 34 58 32 34 98 114 105 100 103 101 34 44 10 32 32 32 32 32 32 34 105 112 97 109 34 58 32 123 10 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 115 116 97 11
6 105 99 34 44 10 32 32 32 32 32 32 32 32 34 114 111 117 116 101 115 34 58 32 91 10 32 32 32 32 32 32 32 32 32 32 123 32 34 100 115 116 34 58 32 34 49 57 50 46 48 46 48 46 48 47 56 34 44 32 34 103 119 34 58 32 34 49 57 50 46 49 54 56 46 53 48 46 49 34 32 125 10 32 32 32 32 32 32 32 32 93 10 32 32 32 32 32 32 125 10 32 32
 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 109 97 99 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 105 112 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 32 32 125 44 10 32 32 
32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 116 117 110 105 110 103 34 10 32 32 32 32 125 10 32 32 93 10 125[]}                                                                                                                                                                                                              
multus-daemon 2025-02-25T14:04:57Z [debug] mergeCNIRuntimeConfig: <nil> &{{0.3.1 iot-vlan  map[] {} {[]  [] []} map[] <nil> []} {0.3.1 iot-vlan false [0xc000399040 0xc000399110]} network/iot-vlan  f1:fd:59:0b:bd:2b  [192.168.50.8/24] [] <nil> <nil> false false false true   [123 10 32 32 34 99 110 105 86 101 114 115 105 1
1 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 34 110 97 109 101 34 58 32 34 105 111 116 45 118 108 97 110 34 44 10 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 100 101 98 117 103 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 109 9
7 99 118 108 97 110 34 44 10 32 32 32 32 32 32 34 109 97 115 116 101 114 34 58 32 34 101 116 104 48 46 53 48 34 44 10 32 32 32 32 32 32 34 109 111 100 101 34 58 32 34 98 114 105 100 103 101 34 44 10 32 32 32 32 32 32 34 105 112 97 109 34 58 32 123 10 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 115 116 97 116 1
05 99 34 44 10 32 32 32 32 32 32 32 32 34 114 111 117 116 101 115 34 58 32 91 10 32 32 32 32 32 32 32 32 32 32 123 32 34 100 115 116 34 58 32 34 49 57 50 46 48 46 48 46 48 47 56 34 44 32 34 103 119 34 58 32 34 49 57 50 46 49 54 56 46 53 48 46 49 34 32 125 10 32 32 32 32 32 32 32 32 93 10 32 32 32 32 32 32 125 10 32 32 32
 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 109 97 99 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 105 112 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 32 32 125 44 10 32 32 32 
32 32 32 32 32 34 116 121 112 101 34 58 32 34 116 117 110 105 110 103 34 10 32 32 32 32 125 10 32 32 93 10 125[]}                                                                                                                                                                                                                 
multus-daemon 2025-02-25T14:04:57Z [debug] mergeCNIRuntimeConfig: add runtimeConfig for net-attach-def: {[] <nil> []    }                                                                                                                                                                                                         
multus-daemon 2025-02-25T14:04:57Z [debug] mergeCNIRuntimeConfig: add runtimeConfig for net-attach-def: {[] <nil> [192.168.50.8/24] f1:fd:59:0b:bd:2b   }                                                                                                                                                                         
multus-daemon 2025-02-25T14:04:57Z [debug] DelegateDel: &{0xc0000a60a0 /hostroot {}}, &Pod{ObjectMeta:{busybox-6f6c985b77-cvmtk busybox-6f6c985b77- kube-system  cdb2ecf5-f300-4991-ae4c-4176704f765b 411183101 0 2025-02-25 14:04:41 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:busybox app.kubernetes.io/instance:bus
ybox app.kubernetes.io/name:busybox pod-template-hash:6f6c985b77] map[k8s.v1.cni.cncf.io/networks:[{                                                                                                                                                                                                                              
multus-daemon   "name": "iot-vlan",                                                                                                                                                                                                                                                                                               
multus-daemon   "namespace": "network",                                                                                                                                                                                                                                                                                           
multus-daemon   "ips": ["192.168.50.8/24"],                                                                                                                                                                                                                                                                                       
multus-daemon   "mac": "f1:fd:59:0b:bd:2b"                                                                                                                                                                                                                                                                                        
multus-daemon }]                                                                                                                                                                                                                                                                                                                  
multus-daemon ] [{apps/v1 ReplicaSet busybox-6f6c985b77 e7426fa9-9c4e-4a6c-901f-f2bb1bdc336b 0xc0002e5ea7 0xc0002e5ea8}] [] []},Spec:PodSpec{Volumes:[]Volume{},Containers:[]Container{Container{Name:busybox,Image:docker.io/library/busybox:latest,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resourc
es:ResourceRequirements{Limits:ResourceList{memory: {{314572800 0} {<nil>} 300Mi BinarySI},},Requests:ResourceList{cpu: {{50 -3} {<nil>} 50m DecimalSI},memory: {{157286400 0} {<nil>} 150Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMes
sagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,},},RestartPolicy:Always,TerminationGracePeriodSec
onds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:dmf-intel-002,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,SupplementalGroup
s:[],FSGroup:*0,RunAsGroup:*0,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:*true,Tolerations:[]Toleration{Toleration{K
ey:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:
[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*false,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,HostUsers:nil,SchedulingGates:[]PodSchedulingGate{},ResourceClai
ms:[]PodResourceClaim{},},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:PodReadyToStartContainers,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:
00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [busybox],},PodCondition{Type:ContainersRe
ady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [busybox],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2025-02-25 14:04:41 +0000 UT
C,Reason:,Message:,},},Message:,Reason:,HostIP:192.168.30.42,PodIP:,StartTime:2025-02-25 14:04:41 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:busybox,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerSt
ate{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:docker.io/library/busybox:latest,ImageID:,ContainerID:,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:Burstable,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuse
s:[]ContainerStatus{},Resize:,ResourceClaimStatuses:[]PodResourceClaimStatus{},HostIPs:[]HostIP{HostIP{IP:192.168.30.42,},},},}, &{{0.3.1 iot-vlan  map[] {} {[]  [] []} map[] <nil> []} {0.3.1 iot-vlan false [0xc000399040 0xc000399110]} network/iot-vlan  f1:fd:59:0b:bd:2b  [192.168.50.8/24] [] <nil> <nil> false false fals
e true   [123 10 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 34 110 97 109 101 34 58 32 34 105 111 116 45 118 108 97 110 34 44 10 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 100 101 98 117 103 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123
10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 32 32 34 109 97 115 116 101 114 34 58 32 34 101 116 104 48 46 53 48 34 44 10 32 32 32 32 32 32 34 109 111 100 101 34 58 32 34 98 114 105 100 103 101 34 44 10 32 32 32 32 32 32 34 105 112 97 109 34 58 32 123 10 32 32 32 32 32
 32 32 32 34 116 121 112 101 34 58 32 34 115 116 97 116 105 99 34 44 10 32 32 32 32 32 32 32 32 34 114 111 117 116 101 115 34 58 32 91 10 32 32 32 32 32 32 32 32 32 32 123 32 34 100 115 116 34 58 32 34 49 57 50 46 48 46 48 46 48 47 56 34 44 32 34 103 119 34 58 32 34 49 57 50 46 49 54 56 46 53 48 46 49 34 32 125 10 32 32 
32 32 32 32 32 32 93 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 109 97 99 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 105 112 115 34 58 32 116 1
14 117 101 10 32 32 32 32 32 32 32 32 125 44 10 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 116 117 110 105 110 103 34 10 32 32 32 32 125 10 32 32 93 10 125[]}, &{d189dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb /var/run/netns/cni-1b7f5ba8-66b9-c727-122b-b4b66f324437 net1 [[IgnoreUnknown true] 
K8S_POD_NAMESPACE kube-system[] [K8S_POD_NAME busybox-6f6c985b77-cvmtk] [K8S_POD_INFRA_CONTAINER_ID d189dd14eeb850fc3eb7902607f8fa65de57ba2b3b073cde9f3871f0789dcedb] [K8S_POD_UID cdb2ecf5-f300-4991-ae4c-4176704f765b]] map[ips:[192.168.50.8/24] mac:f1:fd:59:0b:bd:2b] }

Or anything in here?

Events:                                                            Type     Reason                  Age   From               Message                                                            ----     ------                  ----  ----               -------                                                                 Normal   Scheduled               6s    default-scheduler    Normal   AddedInterface          5s    multus             Add eth0 [172.22.3.236/32] from cilium                                 Warning  FailedCreatePodSandBox  5s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "316a49823b12cdce5a00432825ecff0523d4f16f21c181921e9e6225d498c3f0": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request fa
iled with status 400: 'ContainerID:"316a49823b12cdce5a00432825ecff0523d4f16f21c181921e9e6225d498c3f0" Netns:"/var/run/netns/cni-edefcc22-7b55-8783-45e4-9d20174f1445" IfName:"eth0" Args:"K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=busybox-6f6c985b77-rks6r;K8S_POD_INFRA_CONTAINER_ID=316a49823b12cdce5a00432825ecff0523d4f16f2
1c181921e9e6225d498c3f0;K8S_POD_UID=16d67c4d-f580-4af7-bad1-b7d757a596f7;IgnoreUnknown=1" Path:"" ERRORED: error configuring pod [kube-system/busybox-6f6c985b77-rks6r] networking: [kube-system/busybox-6f6c985b77-rks6r/16d67c4d-f580-4af7-bad1-b7d757a596f7:iot-vlan]: error adding container to network "iot-vlan": plugin typ
e="macvlan" failed (add): failed to create macvlan: cannot assign requested address                                         ': StdinData: {"binDir":"/opt/cni/bin","clusterNetwork":"/host/etc/cni/net.d/05-cilium.conflist","cniVersion":"0.3.1","logLevel":"debug","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"} 

I still can't get this configured correctly.

[dmfrey]

I was able to resolve my issue. It seems to be related to trying to specify MAC addresses in my annotations. I removed them and pods now have the second network.