Skip to content
This repository was archived by the owner on Jun 23, 2022. It is now read-only.

pkg/certsigner/signer: Add "client" usage to server profile #22

Merged
merged 1 commit into from
Jan 21, 2019

Conversation

wking
Copy link
Contributor

@wking wking commented Dec 6, 2018

Avoid issues like:

WARNING: 2018/05/29 11:17:10 Failed to dial 127.0.0.1:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

In the discussion there, the issue seems to be that etcd 3.2 started requiring the client usage for the server cert, which is (for some reason) used when connecting to the gRPC gateway (although I haven't been able to find the etcd code backing that up).

CC @praveenkumar, who saw this issue here.

Avoid issues like [1]:

  WARNING: 2018/05/29 11:17:10 Failed to dial 127.0.0.1:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

In the discussion there, the issue seems to be that etcd 3.2 started
requiring the client usage for the server cert, which is (for some
reason) used when connecting to the gRPC gateway [2,3].

[1]: etcd-io/etcd#9785 (comment)
[2]: etcd-io/etcd#9785 (comment)
[3]: https://github.com/etcd-io/etcd/blob/v3.3.10/Documentation/dev-guide/api_grpc_gateway.md
@praveenkumar
Copy link

I tested this PR and I can't now see the bad: certificate error during the stop => start of the cluster.

Current state

[root@test1-master-0 core]# crictl logs c6ad91a6e6b2d
2018-12-05 05:26:24.658772 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2018-12-05 05:26:24.659785 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-member-test1-master-0
2018-12-05 05:26:24.659841 I | etcdmain: etcd Version: 3.2.14
2018-12-05 05:26:24.659846 I | etcdmain: Git SHA: fb5cd6f1c
2018-12-05 05:26:24.659850 I | etcdmain: Go Version: go1.8.5
2018-12-05 05:26:24.659853 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-05 05:26:24.659857 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-05 05:26:24.660055 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-12-05 05:26:24.660091 I | embed: peerTLS: cert = /etc/ssl/etcd/system:etcd-peer:test1-etcd-0.tt.testing.crt, key = /etc/ssl/etcd/system:etcd-peer:test1-etcd-0.tt.testing.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true
2018-12-05 05:26:24.662353 I | embed: listening for peers on https://0.0.0.0:2380
2018-12-05 05:26:24.662470 I | embed: listening for client requests on 0.0.0.0:2379
2018-12-05 05:26:24.801427 I | etcdserver: name = etcd-member-test1-master-0
2018-12-05 05:26:24.801458 I | etcdserver: data dir = /var/lib/etcd
2018-12-05 05:26:24.801465 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-05 05:26:24.801469 I | etcdserver: heartbeat = 100ms
2018-12-05 05:26:24.801472 I | etcdserver: election = 1000ms
2018-12-05 05:26:24.801476 I | etcdserver: snapshot count = 100000
2018-12-05 05:26:24.801487 I | etcdserver: advertise client URLs = https://192.168.126.11:2379
2018-12-05 05:26:25.044835 I | etcdserver: restarting member 7d3fdaaceb134d3d in cluster d98ef57fc5131193 at commit index 15764
2018-12-05 05:26:25.048030 I | raft: 7d3fdaaceb134d3d became follower at term 2
2018-12-05 05:26:25.048142 I | raft: newRaft 7d3fdaaceb134d3d [peers: [], term: 2, commit: 15764, applied: 0, lastindex: 15764, lastterm: 2]
2018-12-05 05:26:25.095331 W | auth: simple token is not cryptographically signed
2018-12-05 05:26:25.102974 I | etcdserver: starting server... [version: 3.2.14, cluster version: to_be_decided]
2018-12-05 05:26:25.114419 I | etcdserver/membership: added member 7d3fdaaceb134d3d [https://test1-etcd-0.tt.testing:2380] to cluster d98ef57fc5131193
2018-12-05 05:26:25.114542 N | etcdserver/membership: set the initial cluster version to 3.2
2018-12-05 05:26:25.114619 I | etcdserver/api: enabled capabilities for version 3.2
2018-12-05 05:26:25.115735 I | embed: ClientTLS: cert = /etc/ssl/etcd/system:etcd-server:test1-etcd-0.tt.testing.crt, key = /etc/ssl/etcd/system:etcd-server:test1-etcd-0.tt.testing.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true
2018-12-05 05:26:25.649011 I | raft: 7d3fdaaceb134d3d is starting a new election at term 2
2018-12-05 05:26:25.649294 I | raft: 7d3fdaaceb134d3d became candidate at term 3
2018-12-05 05:26:25.649453 I | raft: 7d3fdaaceb134d3d received MsgVoteResp from 7d3fdaaceb134d3d at term 3
2018-12-05 05:26:25.649690 I | raft: 7d3fdaaceb134d3d became leader at term 3
2018-12-05 05:26:25.649856 I | raft: raft.node: 7d3fdaaceb134d3d elected leader 7d3fdaaceb134d3d at term 3
2018-12-05 05:26:25.653658 I | etcdserver: published {Name:etcd-member-test1-master-0 ClientURLs:[https://192.168.126.11:2379]} to cluster d98ef57fc5131193
2018-12-05 05:26:25.654254 I | embed: ready to serve client requests
2018-12-05 05:26:25.662969 I | embed: serving client requests on [::]:2379
WARNING: 2018/12/05 05:26:25 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

After this PR

[root@test1-master-0 core]# crictl logs a57da8e6404b8 => etcd-member container
2018-12-07 05:10:13.016237 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2018-12-07 05:10:13.016972 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-member-test1-master-0
2018-12-07 05:10:13.017089 I | etcdmain: etcd Version: 3.2.14
2018-12-07 05:10:13.017135 I | etcdmain: Git SHA: fb5cd6f1c
2018-12-07 05:10:13.017174 I | etcdmain: Go Version: go1.8.5
2018-12-07 05:10:13.017211 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-07 05:10:13.017248 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2018-12-07 05:10:13.017467 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-12-07 05:10:13.017537 I | embed: peerTLS: cert = /etc/ssl/etcd/system:etcd-peer:test1-etcd-0.tt.testing.crt, key = /etc/ssl/etcd/system:etcd-peer:test1-etcd-0.tt.testing.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true
2018-12-07 05:10:13.019754 I | embed: listening for peers on https://0.0.0.0:2380
2018-12-07 05:10:13.020004 I | embed: listening for client requests on 0.0.0.0:2379
2018-12-07 05:10:13.184095 I | etcdserver: name = etcd-member-test1-master-0
2018-12-07 05:10:13.184199 I | etcdserver: data dir = /var/lib/etcd
2018-12-07 05:10:13.184244 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-07 05:10:13.184291 I | etcdserver: heartbeat = 100ms
2018-12-07 05:10:13.184322 I | etcdserver: election = 1000ms
2018-12-07 05:10:13.184353 I | etcdserver: snapshot count = 100000
2018-12-07 05:10:13.184401 I | etcdserver: advertise client URLs = https://192.168.126.11:2379
2018-12-07 05:10:13.550212 I | etcdserver: restarting member 7d3fdaaceb134d3d in cluster d98ef57fc5131193 at commit index 27829
2018-12-07 05:10:13.559404 I | raft: 7d3fdaaceb134d3d became follower at term 2
2018-12-07 05:10:13.559446 I | raft: newRaft 7d3fdaaceb134d3d [peers: [], term: 2, commit: 27829, applied: 0, lastindex: 27829, lastterm: 2]
2018-12-07 05:10:13.568038 I | mvcc: restore compact to 21992
2018-12-07 05:10:13.591724 W | auth: simple token is not cryptographically signed
2018-12-07 05:10:13.600503 I | etcdserver: starting server... [version: 3.2.14, cluster version: to_be_decided]
2018-12-07 05:10:13.611537 I | etcdserver/membership: added member 7d3fdaaceb134d3d [https://test1-etcd-0.tt.testing:2380] to cluster d98ef57fc5131193
2018-12-07 05:10:13.611907 N | etcdserver/membership: set the initial cluster version to 3.2
2018-12-07 05:10:13.611950 I | embed: ClientTLS: cert = /etc/ssl/etcd/system:etcd-server:test1-etcd-0.tt.testing.crt, key = /etc/ssl/etcd/system:etcd-server:test1-etcd-0.tt.testing.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true
2018-12-07 05:10:13.611984 I | etcdserver/api: enabled capabilities for version 3.2
2018-12-07 05:10:13.760873 I | raft: 7d3fdaaceb134d3d is starting a new election at term 2
2018-12-07 05:10:13.761034 I | raft: 7d3fdaaceb134d3d became candidate at term 3
2018-12-07 05:10:13.761223 I | raft: 7d3fdaaceb134d3d received MsgVoteResp from 7d3fdaaceb134d3d at term 3
2018-12-07 05:10:13.761340 I | raft: 7d3fdaaceb134d3d became leader at term 3
2018-12-07 05:10:13.761462 I | raft: raft.node: 7d3fdaaceb134d3d elected leader 7d3fdaaceb134d3d at term 3
2018-12-07 05:10:13.762732 I | etcdserver: published {Name:etcd-member-test1-master-0 ClientURLs:[https://192.168.126.11:2379]} to cluster d98ef57fc5131193
2018-12-07 05:10:13.763099 I | embed: ready to serve client requests
2018-12-07 05:10:13.772033 I | embed: serving client requests on [::]:2379

@LalatenduMohanty
Copy link

@wking Wondering why this PR is not getting merged? IMO it should be merged asap as it fixes the crucial cert issue in libvirt.

@wking
Copy link
Contributor Author

wking commented Dec 13, 2018

@abhinavdahiya, can you review or delegate to someone who can?

@wking
Copy link
Contributor Author

wking commented Jan 7, 2019

Ping @ericavonb (I hear you're helping with etcd certs in this repo ;).

@mrogers950
Copy link

@wking @ericavonb I'm OK with this PR if this is now a etcd requirement. Fundamentally I don't like the dual-use thing, and it is a bit annoying that this seemed to be an undocumented change, but there isn't much else we can do other than fix etcd to not require dual-usage certs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants