IP Address SAN broken for TLS comms between nodes under 3.2.x

[robertglen]

Bootstrapping a 3.2.1 etcd cluster was giving me endless errors like this:

Jul 03 21:54:45 ip-10-xxx-xxx-xxx etcd[3213]: rejected connection from “10.yyy.yyy.yyy:37194” (tls: “10.yyy.yyy.yyy” does not match any of DNSNames [“*.etcd.ec2.xyz” “etcd.ec2.xyz” “etcd0” “etcd0.ec2.xyz” “etcd1” “etcd1.ec2.xyz” “etcd2” “etcd2.ec2.xyz” “etcd3” “etcd3.ec2.xyz” “etcd4” “etcd4.ec2.xyz” “etcd5” “etcd5.ec2.xyz”])

It complains of the IP address 10.yyy.yyy.yyy not being amongst the DNSNames of the TLS cert provided by 10.yyy.yyy.yyy. Meanwhile 10.yyy.yyy.yyy x509 cert does have its address in the IP Address section of the cert's SAN.

This is new behavior. I simply swapped out the 3.2.1 etcd binary for the newest 3.1.x (3.1.9) and restarted etcd on all 5 nodes, now the cluster is fully operational.

[gyuho]

“10.yyy.yyy.yyy” does not match any of DNSNames [“*.etcd.ec2.xyz” “etcd.ec2.xyz” “etcd0” “etcd0.ec2.xyz” “etcd1” “etcd1.ec2.xyz” “etcd2” “etcd2.ec2.xyz” “etcd3” “etcd3.ec2.xyz” “etcd4” “etcd4.ec2.xyz” “etcd5” “etcd5.ec2.xyz”])

Does any of the hosts resolve to 10.yyy.yyy.yyy?

[robertglen]

No. I have NO DNS records matching any of these names. In my case, I have a script that generates an overloaded cert that could work for any of the hosts if I addressed them via DNS names, but I instead refer to all hosts by IP address in the etcd cmdline args. The script creates an x509 cert with all the names as you see here PLUS an IP Address SAN for 127.0.0.1 and the node's own IP address.

So each node technically has a unique cert with its own IP plus the collection of DNS SANs which as I said are not being used.

Team, I have 3 masters of which one was manually deleted, and it was brought up by auto scale group however, etcd started to throw below on the new master that joined cluster. Any hint?

etcdmain: the server is already initialized as member before, starting as etcd member...
2019-03-21 16:32:25.541529 I | embed: peerTLS: cert = /srv/kubernetes/etcd.pem, key = /srv/kubernetes/etcd-key.pem, ca = , trusted-ca = /srv/kubernetes/ca.crt, client-cert-auth = true
2019-03-21 16:32:25.542119 I | embed: listening for peers on https://0.0.0.0:2380
2019-03-21 16:32:25.542161 I | embed: listening for client requests on 0.0.0.0:4001
2019-03-21 16:32:25.554167 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380
2019-03-21 16:32:25.564718 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380
2019-03-21 16:32:25.649568 I | etcdmain: rejected connection from "11.4.0.4:40736" (error "tls: "11.4.0.4" does not match any of DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"] (lookup etcd-events-c.internal.test.com on 100.64.0.10:53: dial udp 100.64.0.10:53: operation was canceled)", ServerName "etcd-b.internal.test.com", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"])
2019-03-21 16:32:25.683192 C | etcdmain: member 16acea4929706d3 has already been bootstrapped

the etcd.pem shows below
X509v3 Subject Alternative Name:
DNS:etcd-a.internal.test.com, DNS:etcd-b.internal.test.com, DNS:etcd-c.internal.test.com, DNS:etcd-events-a.internal.test.com, DNS:etcd-events-b.internal.test.com, DNS:etcd-events-c.internal.test.com, DNS:localhost, IP Address:127.0.0.1