bootstrap role grants access to kube-system.kube-proxy configmap (and shouldn't?)

[anguslees]

Is this a request for help?

This is a "wat?" that might indicate an area for improvement.

Is this a BUG REPORT or FEATURE REQUEST?

I guess BUG REPORT?

Versions

kubeadm version (use kubeadm version): 1.19.2 release binary

Environment:

• Kubernetes version (use kubectl version): 1.18.6
• Cloud provider or hardware configuration: on-prem / self-hosted(*)
• OS (e.g. from /etc/os-release): Flatcar (aka CoreOS container linux)
• Kernel (e.g. uname -a): 5.4.62-flatcar
• Others:

(*) The server-side is a self-hosted cluster that has deviated from a "normal" kubeadm cluster. Hence why I even noticed this issue.

What happened?

I use kubeadm only for node join operations (TLS bootstrap; ie kubeadm token create && join). I tried to join a new node last week, after not having done so for some time (years?).

What you expected to happen?

I expected kubeadm join --discovery-token-ca-cert-hash=... --token=... --ignore-preflight-errors=all on the new node to create the node bootstrap kubeconfig and CSR. Instead it reported an RBAC error about not being able to read kube-system.kube-proxy ConfigMap (I presume for a preflight check?), and I could not find a way to disable this. By experiment, it seems this requirement crept in with kubeadm 1.13 (I tried every minor release, and kubeadm 1.12 worked).

I can of course add the RBAC role/rolebinding to allow the node bootstrapper group to read this ConfigMap. That seems weird though, which is the point of this bug report.

Why does the node bootstrapper need access to kube-proxy ConfigMap? If it's just for a preflight check of some sort, why can't I skip it? (.. and is increasing the node bootstrapper attack surface really the right tradeoff, just to enable a preflight check?)

How to reproduce it (as minimally and precisely as possible)?

Remove system:bootstrappers:kubeadm:default-node-token group from kube-system.kube-proxy role binding. Try to complete a successful kubeadm join with any amount of flags.

For reference, this bootstrappers role binding is normally added in the proxy addon phase: https://github.com/kubernetes/kubernetes/blob/f2e3154a140dc192760a5cf8c01f8044e1aa867b/cmd/kubeadm/app/phases/addons/proxy/proxy.go#L176-L211

Anything else we need to know?

My kube master config (flags, etc) is visible https://github.com/anguslees/k8s-home/blob/master/kube-system.jsonnet - if it's at all useful. Kubelet flags are https://github.com/anguslees/k8s-home/blob/master/coreos-pxe-install.jsonnet#L169-L227

[fabriziopandini]

@anguslees thanks for this report!
I agree that we should avoid having tokens access the kube-proxy

There is in roadmap a change in v1beta3 that should provide a cleaner solution for users to opt-out from installing kube-proxy, and thus allow to clean up the existing RBAC rule.
I'll defer to @neolit123 to decide if something can be changed before the new release is implemented given that he has more context than me in all the discussion that lead to the introduction of this RBAC rule

[neolit123]

thanks for logging the ticket, i did some deeper investigation here.

do you have kube-proxy running in the cluster - i.e. is the kube-proxy config map present?

kubeadm version (use kubeadm version): 1.19.2 release binary

i'm curious what is the actual error you are seeing on the command line?

we do have an e2e test that was added in 1.19 that ensures that kubeadm tolerates a missing kube-proxy CM (this also skips the RBAC setup):
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-no-addons-master

removing this permission means that fetching the kube-proxy config will not be possible on joining nodes.
but newer versions of kubeadm actually tolerate both missing and forbidden config map:
https://github.com/kubernetes/kubernetes/blob/f2e3154a140dc192760a5cf8c01f8044e1aa867b/cmd/kubeadm/app/componentconfigs/kubeproxy.go#L54

the above logic saw refactors, but in was there in earlier versions too, so the join process should not fail in the support skew 1.17+.

having said that, the kube-proxy configuration matters for the kube-proxy DS and not to the bootstrap token user. unlike the kubelet configuration fetched from the cluster, joining nodes have no use of the kube-proxy config and given it is not part of preflight (at least in newer versions) so there is no need to fetch it at all.

so, a potential much larger refactor here is not fetch the kube-proxy config at all during join.

[neolit123]

having said that, the kube-proxy configuration matters for the kube-proxy DS and not to the bootstrap token user.

merging into #2328 as an optional item.

[anguslees]

the kube-proxy configuration matters for the kube-proxy DS and not to the bootstrap token user.

Right. I don't think it's relevant how it fails, or what I can do to make it succeed[*]. The point of this bug report is that it shouldn't happen at all.

[*] Unless this knowledge is useful for working out how to remove the offending code?

[anguslees]

Oh, and since I just need to try again. kubeadm 1.19.9 fails with:

$ sudo ./kubeadm join --node-name=$(cat /etc/machine-id ) kube.lan:6443 --token cfssku.xxxx     --discovery-token-ca-cert-hash sha256:xxxx --discovery-token-ca-cert-hash sha256:xxxx --ignore-preflight-errors=all
[preflight] Running pre-flight checks
        [WARNING CRI]: container runtime is not running: output: Client:
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
, error: exit status 1
        [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
        [WARNING Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
        [WARNING IsDockerSystemdCheck]: cannot execute 'docker info -f {{.CgroupDriver}}': exit status 2
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.10.25-flatcar
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled (as module)
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_PIDS: enabled
        [WARNING SystemVerification]: error verifying Docker info: "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
        [WARNING Hostname]: hostname "6751cbb9e81a4a928510cec6eec02a78" could not be reached
        [WARNING Hostname]: hostname "6751cbb9e81a4a928510cec6eec02a78": lookup 6751cbb9e81a4a928510cec6eec02a78 on 192.168.0.1:53: no such host
        [WARNING KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W0327 03:15:42.752364    3075 configset.go:79] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" is forbidden: User "system:bootstrap:cfssku" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kubelet-config-1.0" is forbidden: User "system:bootstrap:cfssku" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
To see the stack trace of this error execute with --v=5 or higher
$ echo $?
1

[anguslees]

Fwiw, tried the same join command with kubeadm v1.20.5. Same output (and error). Afaics, I need to continue using kubeadm 1.12 for now :/

[neolit123]

error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kubelet-config-1.0" is forbidden: User "system:bootstrap:cfssku" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

"kubelet-config-1.0" is a very oddly versioned.
the name of this CM should be "kubelet-config-1.19" for a 1.19 cluster.

while kube-proxy CM access for the bootstrap token is not needed, access to the kubelet config CM is mandatory.

[anguslees]

Oh interesting. It's highly likely that my kube-system.kubeadm-config is ancient, since I don't use kubeadm to manage cluster upgrades. I really only use kubeadm for token create and join.

kubectl get cm kubeadm-config output:

apiVersion: v1
data:
  ClusterConfiguration: |
    apiVersion: kubeadm.k8s.io/v1beta1
    kind: ClusterConfiguration
    kubernetesVersion: 1.0.0
  MasterConfiguration: |
    api:
      advertiseAddress: kube.lan
      bindPort: 6443
    apiServerCertSANs:
    - kube.lan
    - kube.oldmacdonald.farm
    authorizationModes:
    - Node
    - RBAC
    certificatesDir: /etc/kubernetes/pki
    cloudProvider: ""
    etcd:
      caFile: ""
      certFile: ""
      dataDir: /var/lib/etcd
      endpoints: null
      image: ""
      keyFile: ""
    featureGates:
      SelfHosting: true
    imageRepository: gcr.io/google_containers
    kubernetesVersion: v1.9.10
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    nodeName: cdd1bfabd47f4cc4b9b5c72df6cbf9c2
    token: ""
    tokenTTL: 12h0m0s
    unifiedControlPlaneImage: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2017-10-10T00:09:34Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "220459491"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config
  uid: 4bdae801-ad4f-11e7-bff3-02120902b07c

while kube-proxy CM access for the bootstrap token is not needed, access to the kubelet config CM is mandatory.

Oh. I don't use kubelet configs at all, and don't have any such configmaps :(
Why does TLS bootstrap require a kubelet config?

[neolit123]

this is breaking some kubeadm assumptions, so join and other operations will fail.

you can create a test kubeadm clusters with a newer version to see what config-maps are written and migrate this old cluster somehow.

Why does generating the bootstrap token require a kubelet config?

joining worker and control-plane nodes download a kubelet configuration from the cluster to configure the local kubelet.

[anguslees]

Ack. So it's possible this bug is really a feature request for a "just join" subcommand. (I already have the kubelet config provided via other means after the kubelet can auth to apiserver)

Or alternatively, you tell me to go away, and I go write my own such tool 😄. I understand that I'm not a normal kubeadm cluster at this point, so my special needs shouldn't be your problem.