Master node labelling inconsistent (label `kubernetes.io/role: master` missing)

[maxieckert-sap]

Describe the bug
When creating a cluster, the label kubernetes.io/role: master is missing on the master nodes in some cases:

Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_E2s_v3
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=westeurope
                    failure-domain.beta.kubernetes.io/zone=0
                    kubernetes.azure.com/cluster=fw7407kpej-maxtestin1-d1
                    kubernetes.azure.com/role=master
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-master-73918563-0
                    kubernetes.io/os=linux

This happens sporadically, not for every cluster. For most of the clusters (created with exactly the same aks-engine version, kubernetes version and API model) it is present on the master nodes:

Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_E2s_v3
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=westeurope
                    failure-domain.beta.kubernetes.io/zone=0
                    kubernetes.azure.com/cluster=ccl7pxm9i6-maxtestin1-d1
                    kubernetes.azure.com/role=master
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-master-25489055-0
                    kubernetes.io/os=linux
                    kubernetes.io/role=master
                    node-role.kubernetes.io/master=

We rely on this label (kubernetes.io/role: master) for the cluster-autoscaler deployment (we are using the same node selector as in the example here: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-master.yaml#L159).

Why does this happen (randomly)?
The existence of which label can we rely on? kubernetes.azure.com/role=master?

Steps To Reproduce

Impaired cluster with missing label:
ExportedTemplate-deploy-fw7407kpej-maxtestin1-d1-kub 2.zip

Cluster with label present:
ExportedTemplate-deploy-ccl7pxm9i6-maxtestin1-d1-kub 2.zip

Expected behavior
The label kubernetes.io/role: master should be present on all master nodes and the node labelling should be consistent.

AKS Engine version
v0.44.2

Kubernetes version
1.16.1

Additional context

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[mboersma]

These three labels should be applied to each node:

• kubernetes.azure.com/role=master
• kubernetes.io/role=master
• node-role.kubernetes.io/master=

But Kubernetes does not allow the latter two labels to be applied directly on kubelet startup, so AKS Engine uses a script run periodically by systemd that checks for the presence of the first label and applies the others. The latter two were somewhat deprecated, but need to be there for backward compatibility with label selectors such as yours.

It sounds like the systemd unit didn't run or the script itself failed. Could you check the status of the label-nodes.service on one of the nodes that didn't get labeled correctly?

sudo systemctl status label-nodes

[mboersma]

See also #3090. Never mind, it's not relevant.

[maxieckert-sap]

The services does not seem to run on the impaired master vm:

$ sudo systemctl status label-nodes
● label-nodes.service - Label Kubernetes nodes as masters or agents
   Loaded: loaded (/etc/systemd/system/label-nodes.service; static; vendor preset: enabled)
   Active: inactive (dead)

$ sudo systemctl is-active label-nodes
inactive

Looks like it failed on start-up and was never executed again:

$ sudo journalctl -u label-nodes
-- Logs begin at Tue 2020-07-14 16:12:32 UTC, end at Fri 2020-07-17 06:57:24 UTC. --
Jul 14 16:12:32 k8s-master-20267257-0 systemd[1]: Stopped Label Kubernetes nodes as masters or agents.
Jul 14 16:12:32 k8s-master-20267257-0 systemd[1]: Started Label Kubernetes nodes as masters or agents.
Jul 14 16:12:47 k8s-master-20267257-0 bash[4826]: The connection to the server localhost:8080 was refused - did you specify the right host or
Jul 14 16:12:47 k8s-master-20267257-0 systemd[1]: label-nodes.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 16:12:47 k8s-master-20267257-0 systemd[1]: label-nodes.service: Unit entered failed state.
Jul 14 16:12:47 k8s-master-20267257-0 systemd[1]: label-nodes.service: Failed with result 'exit-code'.
Jul 14 16:13:34 k8s-master-20267257-0 systemd[1]: Stopped Label Kubernetes nodes as masters or agents.

Does it only run at startup or shouldn't it run periodically in case of any intermittent failures?

After restarting the service (sudo systemctl start label-nodes), the labels were added correctly.

[mboersma]

The label-nodes.service systemd unit is set to start after kubelet.service starts. It's also set to restart always, so the fact that it didn't here is confusing.

It appears that the service started, but a kubectl label command in label-nodes.sh couldn't contact localhost:8080. That can happen on node startup, because although kubelet is running, the kube-apiserver pod may not be available yet. It often takes a few restarts before the labels are applied (you can see this with the kubectl get nodes command, because the ROLES column will show <None> initially, and then switch to master and agent after label-nodes.sh runs successfully.)

Looks like it failed on start-up and was never executed again

Exactly. I haven't seen this happen before. I'm no expert on systemd, but I wrote this to act as a periodic job that expects to fail sometimes.

[maxieckert-sap]

Apparently the service is not restarting, nor running periodically here. At what frequency is the service normally supposed to be executed?

Is it possible to fix that behaviour so that we can rely on the labels being applied?

[maxieckert-sap]

@mboersma any updates on this?

[chewong]

Looks like the script assumes that the KUBECONFIG is stored in ~/.kube/config? Is that where it is being stored?

[mboersma]

@maxieckert-sap I'm sorry, I was off work last week.

@chewong the current version of label-nodes.sh does that wildcard find command to locate a kubeconfig file and use it explicitly. It's a bit of a hack to search for it that way (a hack I borrowed from upstream Kubernetes scripts), but we can't easily know what user name was specified when creating the cluster. That is we can't hard-code /home/azureuser/.kube/config.

But in AKS Engine v0.44.2, label-nodes.sh was an older version that called kubectl without an argument to specify a kubeconfig, so it used the default, implicit configuration. This broke at some point during 1.18 development, which prompted #2929 to fix it in AKS Engine as I describe above.

Why the script is failing is interesting, but I'm even more curious why systemd isn't restarting it every 60 seconds as it should. I have seen it successfully restart the script several times during this initial node startup window, so I know it's not fundamentally broken. AFAICT we are using the correct unit parameters to avoid the StartLimitBurst issue, and from the logs above I don't see the Failed with result 'start-limit' message that would emit.

I tried to reproduce this yesterday with AKS Engine v0.44.2 and Kubernetes 1.16.1, but couldn't make the labeler fail. I also tried upgrading in-place, but no luck. Is there an AKS Engine cluster template (apimodel.json) that can reproduce this reliably?

[maxieckert-sap]

Hi @mboersma,
unfortunately, there is no AKS Engine cluster template available any more, apart from the already attached resources.
Were you able to make any progress on this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.