The value of "datacenter" for TLS ClientHello in a WAN setup is not dynamic?

[splashx]

Overview of the Issue

We're finishing up deployment of a two DC setup, 5 servers in each DC and we're at the final stage where we're bootstrapping TLS. We have managed to have a healthy status for each DC (LAN) but when it comes to WAN, the two clusters can't talk to each other - gossip works, but grpc doesnt.

We see the following in one of the clusters:

ubuntu@server03:~$ ./consul members -wan
Node                 Address             Status  Type    Build  Protocol  DC          Segment
server01.my-dc1  10.125.81.9:8302    alive   server  1.2.3  2         my-dc1  <all>
server01.my-dc2  10.125.25.133:8302  alive   server  1.2.3  2         my-dc2  <all>
server02.my-dc2  10.125.25.137:8302  alive   server  1.2.3  2         my-dc2  <all>
server03.my-dc1  10.125.81.7:8302    alive   server  1.2.3  2         my-dc1  <all>
server03.my-dc2  10.125.25.135:8302  alive   server  1.2.3  2         my-dc2  <all>
server04.my-dc1  10.125.81.8:8302    alive   server  1.2.3  2         my-dc1  <all>
server04.my-dc2  10.125.25.134:8302  alive   server  1.2.3  2         my-dc2  <all>
server05.my-dc1  10.125.81.6:8302    alive   server  1.2.3  2         my-dc1  <all>
server05.my-dc2  10.125.25.136:8302  alive   server  1.2.3  2         my-dc2  <all>
ubuntu@server03:~$ ./consul members 
Node      Address             Status  Type    Build  Protocol  DC          Segment
server01  10.125.25.133:8301  alive   server  1.2.3  2         my-dc2  <all>
server02  10.125.25.137:8301  alive   server  1.2.3  2         my-dc2  <all>
server03  10.125.25.135:8301  alive   server  1.2.3  2         my-dc2  <all>
server04  10.125.25.134:8301  alive   server  1.2.3  2         my-dc2  <all>
server05  10.125.25.136:8301  alive   server  1.2.3  2         my-dc2  <all>

But:

ubuntu@server03:~$ ./consul catalog nodes 
Node      ID        Address        DC
server01  b71357ef  10.125.25.133  my-dc2
server02  3975db7d  10.125.25.137  my-dc2
server03  714b613f  10.125.25.135  my-dc2
server04  6d9c580a  10.125.25.134  my-dc2
server05  3fdbda37  10.125.25.136  my-dc2
ubuntu@server03:~$ ./consul catalog nodes -datacenter=my-dc1
Error listing nodes: Unexpected response code: 500 (No path to datacenter)

⚠️ It's important to node that we don't actually use my-dc1 / my-dc2 - we use a string in the format [a-z]{1,2}\-[a-z]{1,7}[0-9]+.

Reproduction Steps

⚠️ The certificates were created using Consul 1.4.1 (because we had done several rounds of trial of cert generation earlier without success, but that's unrelated):

• consul_domain: sd.example.com

1. Set up 2 clusters with 5 nodes each
2. Set datacenter config variable to my-dc1 and my-dc2 on the 2 clusters
3. Set server_name to server.my-dc1.sd.example.com on each node in my-dc1
4. Set server_name to server.my-dc2.sd.example.com on each node in my-dc2
5. Create TLS certififcates using consul tls commands from v1.4.1 (or openssl with similar CN, SANs configuration).
6. consul tls ca create -domain=sd.example.com
7. consul tls cert create -dc=my-dc1 -domain=sd.example.com -server (repeat 5 times)
8. consul tls cert create -dc=my-dc2 -domain=sd.example.com -server (repeat 5 times)
9. set ca_file, cert_file key_file variables on each node to have common CA file, and individual/own respective cert and key files
10. start nodes in clusters

NOTE: the certificate creation steps were based on the manual.

Consul info for both Client and Server

We see the following error message on my-dc1 (the ip addresses in 10.125.25.0/24 are from my-dc2)

consul.rpc: failed to read byte: remote error: tls: bad certificate from=10.125.25.134:35610
consul.rpc: failed to read byte: remote error: tls: bad certificate from=10.125.25.136:51210
consul.rpc: failed to read byte: remote error: tls: bad certificate from=10.125.25.137:48500
consul.rpc: failed to read byte: remote error: tls: bad certificate from=10.125.25.133:25610
consul.rpc: failed to read byte: remote error: tls: bad certificate from=10.125.25.135:31410

Whe dug a bit deeper (a.k.a tcpdump) to try to decipher what bad certificate would really mean and we noticed that when nodes from my-dc1 try contact nodes of my-dc2, they are sending a ClientHello message with the server_name value of server.my-dc1.<our_domain> - and obviously that will fail, because the nodes from my-dc2:

• Don't have CN containing server.my-dc1.<our_domain> nor
• Have they subjectAltName containing server.my-dc1.<our_domain>

And thus TLS handshake fails.

To solve this problem we had to reissue all the certificates to contain all dcs in the subjectAltName. This is a problem because for every new added cluster in a new DC we need to reissue all certificates to include that new DC in the subjectAltName.

I suppose this is not the desired behavior - IMHO when contacting an IP address of another DC, consul, when acting as a TLS client, should dynamically change the server_name value to dynamically match server.<dc_name>.<consul_domain>.

Operating system and Environment details

Consul 1.2.3, Ubuntu 16.04

[splashx]

It's important to note that this should not be related to:

verify_server_hostname - If set to true, Consul verifies for all outgoing TLS connections that the TLS certificate presented by the servers matches "server.." hostname. By default, this is false, and Consul does not verify the hostname of the certificate, only that it is signed by a trusted CA. This setting is critical to prevent a compromised client from being restarted as a server and having all cluster state including all ACL tokens and Connect CA root keys replicated to it. This is new in 0.5.1.

We have verify_server_hostname unset (thus false) thus we were falling into:

[...], only that it is signed by a trusted CA. [...]

This sentence is not actually true: we're actually leaving it to the TLS lib to verify (based on all the existing rules x509 dictates). Including, but bot limited to:

• CN or subjetAltName matching server_name
• notBefore and notAfter dates
• Root CA trust
• Key Usage

and so on.

So we are leaving it up to the TLS lib to do its thing, thus this block is probably not something we want to do only if verify_server_hostname is set to true - or?

When verify_server_hostname is set to true, it works as expected.

[hanshasselberg]

Thanks for reporting @splashx! I tried to reproduce your issue, but everything is fine for me, independently from setting verify_server_hostname.

Your conclusion is also not quite right, consul has its own cert checking:

consul/tlsutil/config.go

Lines 232 to 278 in a093af3

	func (c *Config) wrapTLSClient(conn net.Conn, tlsConfig *tls.Config) (net.Conn, error) {
	var err error
	var tlsConn *tls.Conn
	tlsConn = tls.Client(conn, tlsConfig)
	// If crypto/tls is doing verification, there's no need to do
	// our own.
	if tlsConfig.InsecureSkipVerify == false {
	return tlsConn, nil
	}
	// If verification is not turned on, don't do it.
	if !c.VerifyOutgoing {
	return tlsConn, nil
	}
	if err = tlsConn.Handshake(); err != nil {
	tlsConn.Close()
	return nil, err
	}
	// The following is lightly-modified from the doFullHandshake
	// method in crypto/tls's handshake_client.go.
	opts := x509.VerifyOptions{
	Roots: tlsConfig.RootCAs,
	CurrentTime: time.Now(),
	DNSName: "",
	Intermediates: x509.NewCertPool(),
	}
	certs := tlsConn.ConnectionState().PeerCertificates
	for i, cert := range certs {
	if i == 0 {
	continue
	}
	opts.Intermediates.AddCert(cert)
	}
	_, err = certs[0].Verify(opts)
	if err != nil {
	tlsConn.Close()
	return nil, err
	}
	return tlsConn, err
	}

It doesn't check server_name unless verify_server_hostname is set.

So we are leaving it up to the TLS lib to do its thing, thus this block is probably not something we want to do only if verify_server_hostname is set to true - or?

We only want to do it when verify_server_hostname is set. Otherwise your clients might not be able to talk to the server if the certs are not setup correctly.

Is there anything else that could help me reproduce your issue?

[mbag]

Thanks for the quick response @i0rek
I'm working on this with @splashx and here are the concrete steps to reproduce the issue on a machine with docker (18.09.2) and docker-compose (1.20.1,). This was tested on a Fedora, and you might need to tweek the networking to be able to access UI via web browser if you are on Mac as we found out.
I used docker-compose.yml from consul/demo/docker-compose-cluster as starting point but I added configuration options to have clusters in 2 DCs. I also added docker-compose-certs.yml to create TLS certificates for my DCs using docker image consul:1.4.1 as this version comes with tls ca/cert create command out of the box.

Steps to reproduce bad certificate error:

In this example variable verify_server_hostname was set to false as you can see in docker-compose.yml downloaded from the gist. From documentation this means that Consul shouldn't verify hostname:

By default, this is false, and Consul does not verify the hostname of the certificate, only that it is signed by a trusted CA.
Below we will generate certs for 2 datacenters from same CA, and they will not be able to communicate via RPC, despite verify_server_hostname being set to false.

1. Download docker-compose-certs.yml and docker-compose.yml from this gist . Store them in the same directory, as they are using shared volume that is created with volume name in the form currentdir_certs
2. Run docker-compose -f docker-compose-certs.yml
   This will create certificates in the shared volume and used by the servers in the cluster
3. After you have certificates created run docker-compose -f docker-compose.yml up
   Wait until clusters stabilize and elect leaders. I recommend not running in detached mode (docker-compose up -d), as you will not see lines like this appear:

[ERR] consul.rpc: failed to read byte: remote error: tls: bad certificate from=172.23.0.205:60178

4. You should have 2 clusters with 5 nodes running on the network 172.23.0.1/16
   Servers with UI are on IPs
   • https://172.23.0.105:8500 (my-dc1)
   • https://172.23.0.205:8500 (my-dc2)
5. Go to UI on either of those IPs and click on datacenter name in the upper left corner, you should see name of the other DC as well. This means that DCs were able to communicate Serf WAN (If I'm not mistaken). Select this other datacenter and you will get following error in your browser:

500 (The backend responded with an error)

In the docker-compose log you will see lines like these:

[ERR] consul.rpc: failed to read byte: remote error: tls: bad certificate from=172.23.0.205:60178
[ERR] http: Request GET /v1/internal/ui/services?dc=my-dc2, error: No path to datacenter from=172.23.0.1:57488

As the consul.rpc error indicates, it happens at the tls pakcage level. If you turn on Wireshark and set packet capture with appropriate filter (monitor connection on 8300/tcp between bootstrap server in my-dc1 and servers trying to connect from my-dc2)

Steps to have successful deployment of 2 DCs:

1. Stop your docker-compose with ctrl+c
2. Make sure we remove containers just in case
   docker-compose -f docker-compose.yml down -v
3. sed -i 's/"verify_server_hostname": false/"verify_server_hostname": true/' docker-compose.yml
   Turn on server hostname verification
4. docker-compose -f docker-compose.yml up
5. Go to UI, and try navigating between DC from UI. You should be able to see all the information about both datacenters, from both UI (i.e. RPC requests are flowing via established TLS from my-dc1 to my-dc2)
6. You can again inspect packets via Wireshark, and see that this time TLS Client Hello sends correct server_name to Server in other DC.

Other Consul versions

docker-compose.yml has consul:1.2.3 version for cluster nodes, since this was version we initially reported, but you can replace that version with consul:1.4.1 and same thing happens. I didn't test v1.4.2, but I expect it is affected as well.

I hope this time you are able to reproduce the issue. If there are any problems please let me or @splashx now. Many thanks again for looking into this issue.

[hanshasselberg]

Thank you @mbag for your detailed response! I now can reproduce your issue!

You found a bug: disabling verify_server_hostname and setting server_name doesn't work in multi-dc. The interesting bit is, that server_name shouldn't be used for RPC in the code, because it messes with multi-dc as you can see.

Another fix for your setup would be to remove the server_name from your config.

I will think about, how to properly fix this on and report back here.