Find a way to install weave-kube alongside rkt/hyperkube

[bboreham]

TL;DR: hyperkube sets up its CNI files inside a container, while weave-kube mounts directories from the host and installs Weave Net CNI files into those. K8s needs one directory that has the full set of files.

Full detail from user:

"

• We run K8S on CoreOS
• K8S's kubelet is run with the default kubelet wrapper from CoreOS like in https://coreos.com/kubernetes/docs/latest/deploy-master.html#create-the-kubelet-unit
• We use the hyperkube image: quay.io/coreos/hyperkube, which runs k8s in a rocket container and not natively on the machine
• The rocket containers has the folders /opt/cni/bin and /etc/cni/net.d inside with things that are required for CNI (flannel, calico binaries and flannel, loopback config files; most notably the loopback binaries, which seem to be always needed)
• Installing weave-kube creates the pods which create a couple of weave binaries and a 10-weave.conf file on the host machine
• Now, if we mount the CNI folders from the host machine into the container (/opt/cni/bin /etc/cni/net.d), it fails, since k8s looks like specific binaries (such as "loopback"), which are not there due to the mount
• We cannot symlink the binaries directly since they need to exist when the kubelet is created (Since the mount points need to exsist)
• If we created empty weave-net files and mounted them, the weave-kube script would not overwrite them with the real ones since the script checks if they already exists

This leads to a sort of chicken-and-egg problem: We are not able to get the weave binaries in the same folder with the other binaries that k8s needs for CNI to work.

Some solutions we could think of:

• Create our own hyperkube image with weave binaries, but we would not like to maintain this
• Download CNI binaries when launching the k8s controller into /opt/cni/bin, but this requires that we keep track of the current uuid specified in the hyperkube's Makefile"

[bboreham]

cc @errordeveloper - any ideas?

[bboreham]

Possibly we can use a kubelet option - kubelet will always look in /opt/cni/bin for the loopback plugin, while a different code path will follow --cni-bin-dir to look for 3rd-party plugins.

[awh]

This issue was moved to weaveworks/weave#2613