[BUG] Cilium + Multus bridge networking broken

[usrbinkat]

Add Multus via cluster network addons operator

Tasks

• add cnao operator
• cnao config
• add br0 resource
• fix cilium strict mode + multus linux bridge bug
• fix cilium cni missing bridge, macvlan, macvtap, etc cni binaries

Reference

• example helm values: ccio/helm/charts/cnao

[usrbinkat]

cnao is mostly developed and committed to main

need to debug issue w/ multi-home pods, cilium, multus, and kubevirt.

Best lead so far: cilium/cilium#20129 (comment)

[usrbinkat]

Resolve issue with multus functionality on Talos using daemonset patching /run/netns to /var/run/netns to accomodate Talos's read only root filesystem.