dns discovery: set initial-advertise-peer-urls as https url

[zyf0330]

I use nginx to proxy etcd server and do ssl termination, it use http2. Etcd uses DNS Discovery. And after I startup, cluster cannot work normally.
This my etcd startup cmd

etcd --name thor{num} --data-dir /var/lib/etcd --initial-cluster-token etcd-cluster --initial-cluster-state new --discovery-srv example.com --initial-advertise-peer-urls https://thor01.example.com:4760 --listen-peer-urls http://127.0.0.1:2380 --advertise-client-urls https://thor01.example.com:4758 --listen-client-urls http://127.0.0.1:2379

log

2017-08-24 15:47:09.139152 I | etcdmain: etcd Version: 3.2.6
2017-08-24 15:47:09.139187 I | etcdmain: Git SHA: 9d43462
2017-08-24 15:47:09.139190 I | etcdmain: Go Version: go1.8.3
2017-08-24 15:47:09.139195 I | etcdmain: Go OS/Arch: linux/amd64
2017-08-24 15:47:09.139198 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2017-08-24 15:47:09.139246 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-08-24 15:47:09.139307 I | embed: listening for peers on http://127.0.0.1:2380
2017-08-24 15:47:09.139337 I | embed: listening for client requests on 127.0.0.1:2379
2017-08-24 15:47:09.146167 I | etcdserver: name = thor01
2017-08-24 15:47:09.146180 I | etcdserver: data dir = /var/lib/etcd
2017-08-24 15:47:09.146185 I | etcdserver: member dir = /var/lib/etcd/member
2017-08-24 15:47:09.146188 I | etcdserver: heartbeat = 100ms
2017-08-24 15:47:09.146191 I | etcdserver: election = 1000ms
2017-08-24 15:47:09.146194 I | etcdserver: snapshot count = 100000
2017-08-24 15:47:09.146204 I | etcdserver: advertise client URLs = https://101.251.220.234:4758
2017-08-24 15:47:09.147013 I | etcdserver: restarting member f5e2bce5ae996f7 in cluster 771871d99223fdd1 at commit index 3
2017-08-24 15:47:09.147049 I | raft: f5e2bce5ae996f7 became follower at term 15
2017-08-24 15:47:09.147058 I | raft: newRaft f5e2bce5ae996f7 [peers: [], term: 15, commit: 3, applied: 0, lastindex: 3, lastterm: 1]
2017-08-24 15:47:09.149173 W | auth: simple token is not cryptographically signed
2017-08-24 15:47:09.150181 I | etcdserver: starting server... [version: 3.2.6, cluster version: to_be_decided]
2017-08-24 15:47:09.150888 I | etcdserver/membership: added member f5e2bce5ae996f7 [http://thor01.example.com:4760] to cluster 771871d99223fdd1
2017-08-24 15:47:09.150975 I | etcdserver/membership: added member fea6d1bf0db64b9 [http://thor02.example.com:4760] to cluster 771871d99223fdd1
2017-08-24 15:47:09.150998 I | rafthttp: starting peer fea6d1bf0db64b9...
2017-08-24 15:47:09.151021 I | rafthttp: started HTTP pipelining with peer fea6d1bf0db64b9
2017-08-24 15:47:09.151254 I | rafthttp: started streaming with peer fea6d1bf0db64b9 (writer)
2017-08-24 15:47:09.151932 I | rafthttp: started streaming with peer fea6d1bf0db64b9 (writer)
2017-08-24 15:47:09.153172 I | rafthttp: started peer fea6d1bf0db64b9
2017-08-24 15:47:09.153194 I | rafthttp: added peer fea6d1bf0db64b9
2017-08-24 15:47:09.153221 I | rafthttp: started streaming with peer fea6d1bf0db64b9 (stream Message reader)
2017-08-24 15:47:09.153237 I | rafthttp: started streaming with peer fea6d1bf0db64b9 (stream MsgApp v2 reader)
2017-08-24 15:47:09.153266 I | etcdserver/membership: added member 3aea22b89d7c833a [http://thor03.example.com:4760] to cluster 771871d99223fdd1
2017-08-24 15:47:09.153280 I | rafthttp: starting peer 3aea22b89d7c833a...
2017-08-24 15:47:09.153308 I | rafthttp: started HTTP pipelining with peer 3aea22b89d7c833a
2017-08-24 15:47:09.153533 I | rafthttp: started streaming with peer 3aea22b89d7c833a (writer)
2017-08-24 15:47:09.154117 I | rafthttp: started streaming with peer 3aea22b89d7c833a (writer)
2017-08-24 15:47:09.154804 I | rafthttp: started peer 3aea22b89d7c833a
2017-08-24 15:47:09.154825 I | rafthttp: added peer 3aea22b89d7c833a
2017-08-24 15:47:09.154839 I | rafthttp: started streaming with peer 3aea22b89d7c833a (stream Message reader)
2017-08-24 15:47:09.154929 I | rafthttp: started streaming with peer 3aea22b89d7c833a (stream MsgApp v2 reader)
2017-08-24 15:47:10.047389 I | raft: f5e2bce5ae996f7 is starting a new election at term 15
2017-08-24 15:47:10.047454 I | raft: f5e2bce5ae996f7 became candidate at term 16
2017-08-24 15:47:10.047467 I | raft: f5e2bce5ae996f7 received MsgVoteResp from f5e2bce5ae996f7 at term 16
2017-08-24 15:47:10.047476 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to fea6d1bf0db64b9 at term 16
2017-08-24 15:47:10.047515 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to 3aea22b89d7c833a at term 16
2017-08-24 15:47:11.147341 I | raft: f5e2bce5ae996f7 is starting a new election at term 16
2017-08-24 15:47:11.147371 I | raft: f5e2bce5ae996f7 became candidate at term 17
2017-08-24 15:47:11.147382 I | raft: f5e2bce5ae996f7 received MsgVoteResp from f5e2bce5ae996f7 at term 17
2017-08-24 15:47:11.147390 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to fea6d1bf0db64b9 at term 17
2017-08-24 15:47:11.147412 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to 3aea22b89d7c833a at term 17
2017-08-24 15:47:12.147331 I | raft: f5e2bce5ae996f7 is starting a new election at term 17
2017-08-24 15:47:12.147369 I | raft: f5e2bce5ae996f7 became candidate at term 18
2017-08-24 15:47:12.147383 I | raft: f5e2bce5ae996f7 received MsgVoteResp from f5e2bce5ae996f7 at term 18
2017-08-24 15:47:12.147396 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to fea6d1bf0db64b9 at term 18
2017-08-24 15:47:12.147408 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to 3aea22b89d7c833a at term 18
2017-08-24 15:47:13.747326 I | raft: f5e2bce5ae996f7 is starting a new election at term 18
2017-08-24 15:47:13.747360 I | raft: f5e2bce5ae996f7 became candidate at term 19
2017-08-24 15:47:13.747370 I | raft: f5e2bce5ae996f7 received MsgVoteResp from f5e2bce5ae996f7 at term 19
2017-08-24 15:47:13.747393 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to fea6d1bf0db64b9 at term 19
2017-08-24 15:47:13.747402 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to 3aea22b89d7c833a at term 19
2017-08-24 15:47:14.153415 W | rafthttp: health check for peer fea6d1bf0db64b9 could not connect: invalid character '<' looking for beginning of value
2017-08-24 15:47:14.154961 W | rafthttp: health check for peer 3aea22b89d7c833a could not connect: invalid character '<' looking for beginning of value
2017-08-24 15:47:15.147290 I | raft: f5e2bce5ae996f7 is starting a new election at term 19
2017-08-24 15:47:15.147317 I | raft: f5e2bce5ae996f7 became candidate at term 20
2017-08-24 15:47:15.147327 I | raft: f5e2bce5ae996f7 received MsgVoteResp from f5e2bce5ae996f7 at term 20
2017-08-24 15:47:15.147335 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to fea6d1bf0db64b9 at term 20
2017-08-24 15:47:15.147341 I | raft: f5e2bce5ae996f7 [logterm: 1, index: 3] sent MsgVote request to 3aea22b89d7c833a at term 20
2017-08-24 15:47:16.150528 E | etcdserver: publish error: etcdserver: request timed out
......

As you see, it shows added member f5e2bce5ae996f7 [http://thor01.example.com:4760] to cluster 771871d99223fdd1. I think it should be https.

This is nginx config

server {
        listen       4760 ssl http2;
        server_name  thor01.example.com;

        ssl_certificate fullchain.pem;
        ssl_certificate_key privkey.pem;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        access_log off;

        location / {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_next_upstream off;
                proxy_pass http://127.0.0.1:2380;
       }
}
server {
        listen       4758 ssl http2;
        server_name  thor01.example.com;

        ssl_certificate fullchain.pem;
        ssl_certificate_key privkey.pem;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        access_log off;

        location / {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_next_upstream off;

                proxy_pass http://127.0.0.1:2379;
       }
}

[heyitsanthony]

How are the SRV records configured?

[zyf0330]

dig +noall +answer SRV _etcd-server._tcp.example.com
_etcd-server._tcp.example.com. 1 IN	SRV	0 0 4760 thor01.example.com.
_etcd-server._tcp.example.com. 1 IN	SRV	0 0 4760 thor02.example.com.
_etcd-server._tcp.example.com. 1 IN	SRV	0 0 4760 thor03.example.com.

[heyitsanthony]

try _etcd-server-ssl._tcp.example.com

[zyf0330]

You alert me! Thanks!
After set SRV for ssl, I got this in etcd log

rafthttp: health check for peer 20fdc7247054886c could not connect: x509: certificate is valid for thor01.example.com, not example.com
rafthttp: health check for peer 20fdc7247054886c could not connect: x509: certificate is valid for thor03.example.com, not example.com

My cert is from LetsEncrypt

[heyitsanthony]

the certificates seem to be matching example.com instead of the *.example.com

[zyf0330]

This is cert info from certbot. Cert of LetsEncrypt cannot match wildcard domain.

Certificate Name: thor01.example.com
Domains: thor01.example.com
Expiry Date: 2017-11-22 01:43:00+00:00 (VALID: 89 days)
Certificate Path: ...
Private Key Path: ...

[heyitsanthony]

I believe this is a bug.

Discovery bootstrapping will use discovery-srv (example.com) for the ServerName to avoid a mitm cert attack:

if strings.Contains(clusterStr, "https://") && cfg.PeerTLSInfo.CAFile == "" {
    cfg.PeerTLSInfo.ServerName = cfg.DNSCluster
}

but the machine hosts have subdomain.example.com, so they won't match. It should probably be "*." + cfg.DNSCluster

[zyf0330]

Thanks a lot! So will you fix it ?

[zyf0330]

I have attempted, but Go is not my language and solving this problem needs some considers about the whole project. So I am not able to give a fix.