Extensible discovery for Cloud Auto-Join

[dnephin]

Feature Description

Currently the cloud auto-join feature allows a user to query a cloud provider API using go-discover to discover nodes that are part of a Consul cluster.

This feature works well for some use cases, but has a couple limitations:

• discovery only works with the cloud providers known to go-discovery. Supporting additional providers requires that go-discovery be updated.
• the API queries performed by go-discovery are limited to a single tag, there's no way to do a more sophisticated query with multiple tags or any other properties that may be unique to a particular environment.

In addition to the go-discovery provider, we could allow the discovery of addresses by any external command. Using a value of exec=/usr/local/bin/query-ec2 would run the script at that path. The script would exit zero and return one server per line on success, or exit non-zero and print an error message of up to 1024 bytes to stderr on error.

The script could use the cloud provider CLI, or any other command, to perform the query. Since these operations are already performing the API requests, the overhead of starting a new process shouldn't be too significant.

Use Case(s)

This approach could potentially solve a number of use cases:

• provider/hcloud: Add hcloud discovery go-discover#159 - it would allow cloud auto-join to work with new or less widely used clouds without waiting on the code to be merged into go-discover and then Consul.
• Add support for multiple tags and host labels for cloud auto-join #6946 - it would allow for more complex queries that don't fit into the single-tag model of go-discover.
• It would allow auto-join to work with any custom environment, even those that don't have public APIs or public API clients.

Background

go-discover also accounts for about 30 of the 107Mb used by the Consul binary. It also the library that brings in the most number of dependencies, which can make upgrading to new major versions of shared dependencies challenging, and also causes a lot of false positives on security scans. None of these issues by themselves are all that significant, but together they do add up. An approach that removes the need to include every provider in the binary would help mitigate these problems.

Other related issues #9754

[RickyGrassmuck]

I like this idea and have added some feedback in this comment.

It would be nice if the work on each of the cloud providers done in the go-discover package could be retained by releasing each of them as individual binaries that would be loaded via the mechanism proposed in this issue.

[dnephin]

That's an excellent point. I believe go-discover already provides a binary, so as long as this mechanism allows us to add arguments it should be easy to keep using go-discover as a binary instead of a library. Currently the binary uses spaces to separate addresses in the output. Either we'll need to change that to newlines, or this new feature should accept any whitespace separated addresses.

[sevensolutions]

What about a simple HTTP request which needs to return eg. a json array with IP-addresses to join?
Everyone can implement a simple webserver which serves a static config, forwards the request to Proxmox, vSphere or whatever.