api: loadbalancer: Internal does not create correct Route53 entries

[dcowden]

Operating Environment
kops 1.8.0
aws
k8s 1.8.4

What we expect to happen
Our kops spec looks like this:

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-12-21T14:37:50Z
  name: ecom-2.dev.domain.com
spec:
  api:
    loadBalancer:
      type: Internal
...
  masterInternalName: api.internal.ecom-2.dev.domain.com

Per the documentation here, we expect kops to create and maintain a route53 record for api.internal.ecom-2.dev.domain.com, which points to an AWS loadbalancer created for the masters.

What actually happens
We get instead a Route53 record that is a list of A records, corresponding to the masters .

Why this matters
Our workloads experience failures because the nodes cannot talk to a master any more--probably because they are using the ip address directly. When the node dies, workloads get failures doing lookups using kube-dns-- research has shown that this is due to the fact that kube-dns on the node is timing out trying to get to the masters via the old ip.

We can temporarily fix the issue by manually updating the Route53 record to point to the load balancer. We've verified that if we do this, and THEN kill a master, the workloads are not affected.

The problem is that when the new master comes back on line, the cluster re-writes the Route53 name back to A records-- so the workaround will not survive rolling updates.

I believe this is the root cause for #4247, and #3702, which I am going to close in favor of this.

I think this may be the actual root cause of #2634. The solution mentioned there is exactly as I describe, but I suspect would not survive a rolling update.

What we need

We need to know what configuration we have wrong that causes us to get an internal name pointing to the masters directly, and how to fix it.

Alternatively, a workaround that simply disables kops from updating the internal DNS record would be super helpful.

[dcowden]

as a workaround, we updated the api.internal dns record, and then scaled dns-controller to 0 to prevent it from getting changed back.

This resulted an in improvement, but it means now k8s cannot create dns names when we launch services, so its not acceptable long term.

We also noted that while our workloads kept running this time, the nodes in the same AZ as the master still went to NotReady, making them unschedulable. This is still not the desired behavior.

[rlees85]

I've been battling with something similar. The api. DNS record is created correctly as an alias to the load balancer but as soon as any of the masters comes up it is overwritten by 3 IP addresses.

In my instance, masterInternalName and masterPublicName where set the same in the YAML. Changing them to be different fixed the problem at least for masterPublicName.

I doubt it will solve your problem as you seem to be having problems with the k8s nodes talking internally to now-dead masters.

Maybe

  api:
    loadBalancer:

Only supports the public endpoint, which maybe should be documented more clearly.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[zachaller]

Now that this issue has been accepted in kubernetes kubernetes/kubernetes#63492 I feel it should be a valid option to turn on an elb for the internal api to use because I think we are now running into possible timing issues with dns updates and propagating that having an elb there would make it much easier to manage. With the dns propagation taking so long we get nodes going not ready if we say terminate all 3 masters.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[mellowplace]

Looking at the code the problem appears to be the internal annotation is always added to the apiserver pod https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/kube_apiserver.go#L444

This means the record will always be updated by dns-controller. Should this annotation not be conditional in the same way as the external one?

[mellowplace]

Just to chime in with a bit more detail on this. We’ve been affected by a memory leak in kube-apiserver that causes a master node to become unresponsive until the docker health check is able to restart docker (which took 15 mins in one case due to high memory pressure).

As far as I can tell dns-controller does not do any modifications to the internal api record based on the availability of the apiserver pods or nodes so the unresponsive pod remains as one of the IPs in the set.

This in turn means, on occasion, kublet for some worker nodes tries to connect to the struggling master and fails resulting in that node being marked NotReady.

This seems naive on behalf of kubernetes and I suspect some improvements need to be made there but in the meantime we’d also like to use the ELB created by kops for kubelet, kube-proxy etc on worker nodes. At least this way a health check could take care of taking unresponsive apiservers out of the pool.

[mellowplace]

/remove-lifecycle rotten

[mellowplace]

MR with proposed fix here #5375