Providing keys and certs to access etcd

[kit1980]

Is there a way to provide keys and certs to use for accessing the etcd endpoint?
Similar to these CURL options: --cacert /etc/kubernetes/certs/ca.crt --cert /etc/kubernetes/certs/client.crt --key /etc/kubernetes/certs/client.key

python-etcd seems to have ways to setup this, EtcdRendezvous doesn't.

[kit1980]

And a prerequisite to the certs question: etcd_rendezvous.py doesn't seem to provided a way to specify https protocol, right?

[kiukchung]

You are correct, we do not properly dependency inject the EtcdClient into EtcdRendezvous, therefore not allowing the user to pass additional constructor arguments when creating EtcdClient.

We could add a kwargs constructor argument to EtcdRendezvous that we simply expand and pass-through EtcdClient, and construct the kwargs map by parsing any "unknown" additional parameters in the rendezvous URL. For instance:

etcd://localhost:2379/123?min_workers=1&max_workers=8&cacert=/etc/kubernetes/certs/ca.crt&cert=/etc/kubernetes/certs/client.crt&key=/etc/kubernetes/certs/client.key&protocol=https

In this example we could construct the kwargs:

cacert=/etc/kubernetes/certs/ca.crt
cert=/etc/kubernetes/certs/client.crt
key=/etc/kubernetes/certs/client.key
protocol=https

and pass it to the constructor of EtcdClient (

elastic/torchelastic/rendezvous/etcd_rendezvous.py

Line 169 in f5a0968

self.client = etcd.Client(host=endpoints, allow_reconnect=True)

)

class EtcdRendezvous(object):
   def __init__(self, <... existing args...>, kwargs):
        <...omitted for brevity...>
        self.client = etcd.Client(host=endpoints, allow_reconnect=True, **kwargs)

Let me know if that helps and if so I can make these changes. Also PRs are welcome!

[kit1980]

This sounds good to me.
I can try this in my setup and send the PR.

[kiukchung]

Sounds good, looking forward to your PR! You may already know this but I figured this may save you some time, the code below is where we parse the rendezvous URL that is passed by the user:

https://github.com/pytorch/elastic/blob/master/torchelastic/rendezvous/etcd_rendezvous.py#L1126