Rolling update puts nodes into "not ready"

[recollir]

1. Kops version 1.8.0

2. Kubernetes version 1.8.6

3. AWS (3 masters and 3 nodes)

4. kops edit followed by kops update and kops rolling-update. kops edit to add configuration flags for the apiserver (dex related). Also tried kops rolling-update --instance-group <master...> to only update one master at a time.

5. Nodes become "not ready" in an unpredictable way. Sometimes no node is affected. Sometimes one node becomes "not ready" and recovers after a few minutes. Sometimes all nodes are "not ready" for a longer period. Up to 15 minutes. While the masters report ready. During this time the workload on the cluster is not accessible.

6. Nothing: a non-breaking rolling update without affecting nodes or the workload.

7. Starting config: https://gist.github.com/recollir/9e9b4b0b426ef77014083f1839c123d6
   Added via kops edit before the rolliing-update: https://gist.github.com/recollir/da9fd8a123b58f555f2e4321093e9d46

8. https://gist.github.com/recollir/5b19d543adaa50b1889aabafeb77b847

9. A couple of times I observed that after the rolling update the ELB for the API server was missing AZ attached to it.

[johanneswuerbach]

In our case manually restarting the kubelet helped, do you have the logs of an affected node?

[recollir]

Not from the current test runs (cluster has been deleted and created a couple of times). But I can recreate this, keep the logs and attach them to this issue.

Nevertheless, I would be interested how to prevent this from happening. We need to do some changes to a production cluster where a restart of a kubelet seems rather inappropriate.

[johanneswuerbach]

Understandably, we are also started seeing this upgrading to kubernetes 1.8 (1.8.10 currently) and I’m currently debugging what could cause this.

It looks like in our case the kubelet tries to connect to an old API server IP so either its caching the dns resolution somehow too long (TTL should only allow 60s) or the record wasn’t updated correctly.

[recollir]

Thanks for the pointing this out. I will see if this is the same for us first thing in the morning. My TZ is CEST.

[justinsb]

Thanks for reporting & sorry about the problem.

Was this with a gossip DNS (.k8s.local) or a "real" Route53 DNS name?

[johanneswuerbach]

In our case a "real" Route53 record. Also restarting the kubelet almost immediately fixed the issue, while just waiting took up to 15mins for the node to be marked as Ready again.

We are running kops 1.9.0-beta.2.

[recollir]

Real Route53 DNS name.

[recollir]

@johanneswuerbach seems the same for us - kubelet trying to connect to an old API server IP. Trying to verify this now.

[johanneswuerbach]

Could you check whether the internal master DNS contains the IPs of the new masters or is still returning an old one?

[johanneswuerbach]

We also hit this on another node again:

kubelet[1270]: E0410 13:01:58.506984    1270 kubelet_node_status.go:390] Error updating node status, will retry: error getting node "ip-xxx.ec2.internal": Get https://api.internal.xxx/api/v1/nodes/ip-xxx.ec2.internal?resourceVersion=0: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
kubelet[1270]: E0410 13:02:08.507374    1270 kubelet_node_status.go:390] Error updating node status, will retry: error getting node "ip-xxx.ec2.internal": Get https://api.internal.xxx/api/v1/nodes/ip-xxx.ec2.internal: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
and eventually
kubelet[1270]: E0410 13:02:38.508388    1270 kubelet_node_status.go:382] Unable to update node status: update node status exceeds retry count

The IP is the IP of the node itself.

[recollir]

That is exactly the error msg I see. It starts to appear when the old IP address is removed from the A record for api.internal.xxx and the new IP address for the new master is added. Sometimes after the first master, sometimes after the second master.

[lkysow]

It's probably due to: kubernetes/kubernetes#41916 (comment) where the kubelet caches the IP of the old master nodes. That's why a restart fixes it.

[sstarcher]

I had the same issue today with 1.9.0-beta-2

[lkysow]

I think the best practice is to set up an internal ELB that fronts the masters and have the API url point to that, the same way it's done for the external api. Is that possible with kops right now?