More helpful logs if a single-node cluster has nontrivial discovery config

[DaveCTurner]

Users sometimes accidentally auto-bootstrap their nodes into single-node clusters and then try and join them all together later by setting discovery.seed_hosts or similar. Elasticsearch doesn't care if discovery points to a few extra nodes so today it reports no problems, but it's frustrating to the user that it doesn't do what they expect it to do and also doesn't indicate why.

It's almost certainly incorrect to have a nontrivial discovery config in a single-node cluster: discovery.seed_hosts and discovery.seed_providers should both be unset or empty¹, so this is a strong indicator that the user is trying to form a multi-node cluster incorrectly. We can therefore be more helpful to these users by emitting periodic warnings in a single-node cluster with nontrivial discovery config:

This node is a fully-formed single-node cluster with cluster UUID [{clusterUUID}], but [{setting}] is set to [{value}] as if to discover other nodes and form a multi-node cluster. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

Footnotes

1. it's also (kind of) acceptable if discovery.seed_hosts contains just the node's own publish address, or we're using just the file provider and unicast_hosts.txt contains just the node's own publish address. ↩

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[chen-ni]

Is anybody working on this? If not can I take it?

[Leaf-Lin]

Hi, @chen-ni , I have added the label by mistake, sorry for the confusion. This one has been assigned to @kingherc

[chen-ni]

@Leaf-Lin Got it :)

[kingherc]

Hi @DaveCTurner , in your original description you have given a footnote of a kind of valid/acceptable setting for the seed hosts, where the single-node cluster mentions its own public address. Should we take into consideration this acceptable setting? Or should we always emit the periodic warning when the seed hosts are non-empty, and hint to removing the setting (even in the valid/acceptable setting where the single-node cluster mentions itself)?

[DaveCTurner]

Unfortunately yes I think we must ignore our own address before checking for a nonempty seed hosts. In ESS and ECK and so on I believe one-node clusters will be configured to discover themselves.

[kingherc]

@DaveCTurner thanks for the explanation! Finally, do we already store internally the cluster's own public address somewhere, so as to compare? Or should we figure it out programmatically?

[DaveCTurner]

You can get it by calling getAddress() on the DiscoveryNode object corresponding to the local node, which you can typically access via transportService.getLocalNode() or clusterState.getNodes().getLocalNode() (and probably many other paths too).

[kingherc]

Hi @DaveCTurner ! In trying this case of a single-node cluster trying to join another one, I do see a warning already emitted:

[2022-06-22T11:45:31,264][WARN ][o.e.c.c.Coordinator ] [node_t0] failed to validate incoming join request from node [{other0}{KPOWduSVSsy5gNiA9TBbhQ}{HPOV9FTnRfeyU2tjiYkVMQ}{other0}{127.0.0.1}{127.0.0.1:13302}{cdfhilmrstw}]
org.elasticsearch.transport.RemoteTransportException: [other0][127.0.0.1:13302][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: This node previously joined a cluster with UUID [saM90swPSi-BOfsMybowqw] and is now trying to join a different cluster with UUID [3u784dcFRj2bG5P3aRZEIQ]. This is forbidden and usually indicates an incorrect discovery or cluster bootstrapping configuration. Note that the cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path [/tmp/org.elasticsearch.discovery.single.SingleNodeDiscoveryIT_FCD470C364EA96D8-001/tempDir-003/other0] which will also remove any data held by this node.
at org.elasticsearch.cluster.coordination.JoinValidationService.lambda$new$2(JoinValidationService.java:130) ~[classes/:?]
...

So, I do see already some warning message for the use case you mentioned. The warning message is different than the one you suggested, but with similar content. Are you aware of this warning message? Or is it a different use case?