Move metadata storage to Lucene

[ywelsch]

Backport of #50907

Today we split the on-disk cluster metadata across many files: one file for the metadata of each
index, plus one file for the global metadata and another for the manifest. Most metadata updates
only touch a few of these files, but some must write them all. If a node holds a large number of
indices then it's possible its disks are not fast enough to process a complete metadata update before timing out. In severe cases affecting master-eligible nodes this can prevent an election
from succeeding.

This commit uses Lucene as a metadata storage for the cluster state, and is a squashed version
of the following PRs that were targeting a feature branch:

• Introduce Lucene-based metadata persistence (Introduce Lucene-based metadata persistence #48733)

This commit introduces LucenePersistedState which master-eligible nodes
can use to persist the cluster metadata in a Lucene index rather than in
many separate files.

Relates #48701

• Remove per-index metadata without assigned shards (Remove per-index metadata without assigned shards #49234)

Today on master-eligible nodes we maintain per-index metadata files for every
index. However, we also keep this metadata in the LucenePersistedState, and
only use the per-index metadata files for importing dangling indices. However
there is no point in importing a dangling index without any shard data, so we
do not need to maintain these extra files any more.

This commit removes per-index metadata files from nodes which do not hold any
shards of those indices.

Relates #48701

• Use Lucene exclusively for metadata storage (Use Lucene exclusively for metadata storage #50144)

This moves metadata persistence to Lucene for all node types. It also reenables BWC and adds
an interoperability layer for upgrades from prior versions.

This commit disables a number of tests related to dangling indices and command-line tools.
Those will be addressed in follow-ups.

Relates #48701

• Add command-line tool support for Lucene-based metadata storage (Add command-line tool support for Lucene-based metadata storage #50179)

Adds command-line tool support (unsafe-bootstrap, detach-cluster, repurpose, & shard
commands) for the Lucene-based metadata storage.

Relates #48701

• Use single directory for metadata (Use single directory for metadata #50639)

Earlier PRs for #48701 introduced a separate directory for the cluster state. This is not needed
though, and introduces an additional unnecessary cognitive burden to the users.

Co-Authored-By: David Turner david.turner@elastic.co

• Add async dangling indices support (Add async dangling indices support #50642)

Adds support for writing out dangling indices in an asynchronous way. Also provides an option to
avoid writing out dangling indices at all.

Relates #48701

• Fold node metadata into new node storage (Fold node metadata into new node storage #50741)

Moves node metadata to uses the new storage mechanism (see #48701) as the authoritative source.

• Write CS asynchronously on data-only nodes (Write CS asynchronously on data-only nodes #50782)

Writes cluster states out asynchronously on data-only nodes. The main reason for writing out
the cluster state at all is so that the data-only nodes can snap into a cluster, that they can do a
bit of bootstrap validation and so that the shard recovery tools work.
Cluster states that are written asynchronously have their voting configuration adapted to a non
existing configuration so that these nodes cannot mistakenly become master even if their node
role is changed back and forth.

Relates #48701

• Remove persistent cluster settings tool (Remove persistent cluster settings tool #50694)

Adds the elasticsearch-node remove-settings tool to remove persistent settings from the on
disk cluster state in case where it contains incompatible settings that prevent the cluster from
forming.

Relates #48701

• Make cluster state writer resilient to disk issues (Make cluster state writer resilient to disk issues #50805)

Adds handling to make the cluster state writer resilient to disk issues. Relates to #48701

• Omit writing global metadata if no change (Write global meta only when changed #50901)

Uses the same optimization for the new cluster state storage layer as the old one, writing global
metadata only when changed. Avoids writing out the global metadata if none of the persistent
fields changed. Speeds up server:integTest by ~10%.

Relates #48701

• DanglingIndicesIT should ensure node removed first (DanglingIndicesIT should ensure node removed first #50896)

These tests occasionally failed because the deletion was submitted before the
restarting node was removed from the cluster, causing the deletion not to be
fully acked. This commit fixes this by checking the restarting node has been
removed from the cluster.

Co-authored-by: David Turner david.turner@elastic.co

[ywelsch]

@elasticmachine run elasticsearch-ci/2