How to connect to one builtin Etcd cluster(TSL secured) owned by Kubernetes

[leo-lao]

Here I have one k8s cluster,
and I want to access data in its etcd cluster outside k8s cluster(like my local desktop)

• Firstly, I succeeded to access etcd inside the k8s cluster, like this:

etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
get foo

• Then, I copied above files to my own MacPro, and this time it didnot work:

etcdctl --endpoints=10.252.125.xx:2379 \
--cacert=/Users/chenguli/a/etcd/ca.crt \
--cert=/Users/chenguli/a/etcd/healthcheck-client.crt \
--key=/Users/chenguli/a/etcd/healthcheck-client.key \
get foo

reponse message:

{"level":"warn","ts":"2022-02-20T17:39:06.596+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006b8380/10.252.125.84:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded

So My question is, How can I get access to this remote ETCD cluster on my desktop?
—

• image version: registry.aliyuncs.com/k8sxio/etcd:3.4.13-0
• etcd server start up command:

etcd --advertise-client-urls=https://10.252.125.xx:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://10.252.125.xx:2380 --initial-cluster=k8s-master=https://10.252.125.xx:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://10.252.125.xx:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://10.252.125.xx:2380 --name=k8s-master --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

• TLS config files stored on remote etcd cluster:

[root@k8s-node/]# ls -l /etc/kubernetes/pki/etcd
total 36
-rw------- 1 root root 1058 Jun  3  2021 ca.crt
-rw------- 1 root root 1675 Jun  3  2021 ca.key
-rw------- 1 root root 1159 Jun  3  2021 healthcheck-client.crt
-rw------- 1 root root 1679 Jun  3  2021 healthcheck-client.key
-rw------- 1 root root 1708 Feb 20 03:26 healthcheck-client-key.pem
-rw------- 1 root root 1204 Jun  3  2021 peer.crt
-rw------- 1 root root 1679 Jun  3  2021 peer.key
-rw------- 1 root root 1204 Jun  3  2021 server.crt
-rw------- 1 root root 1679 Jun  3  2021 server.key

• Is there a proxy between the client and the etcd cluster?
  I logged onto one specific VPN network(provided by our company), then tried to connect to remote etcd cluster.
  I don't know if this is so called proxy.

[leo-lao]

have searched many pages:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/
https://etcd.io/docs/v3.4/op-guide/security/

I guess this is one TSL related problem?
certificates copied from k8s /etc/kubernetes/pki/etcd folder are not suitablel for remote access?
Maybe one brand new client certificates need to be generated?

Help wanted!

[ahrtr]

Please follow the template to provide all info (such as etcd version) when raising an issue.

[serathius]

@ahrtr, this issue was not filed as bug and author used empty template.

[ahrtr]

OK, got it, thanks @serathius .

@leo-lao Please let us know what's the etcd version? Is there a proxy between the client and the etcd cluster? FYI. 13192

[leo-lao]

@ahrtr Hi, I have updated this issue, and hope this time enough information are provided.

[ahrtr]

Your etcd version is 3.4.13, so it doesn't have the issue 13192.

Your issue could be just a network issue. Please verify the connectivity firstly using tools something like openssl, telnet, curl, etc. The key point is to make sure the IP:Port is reachable.

Also set the endponts as https://10.252.125.xx:2379

[leo-lao]

@ahrtr thanks for you kind answer.

I tried netcat 10.252.125.xx:2379, and  found can't connect to it.
this is indeed one connection problem, but currently I still can't find out the reason.
anyway, thank you very much.
I will dig into this issue on my own.