Replicate closed indices

[tlrx]

Closed indices currently have the downside that they are not replicated, so it's possible for an index to be closed, then the data lost because the node was terminated without saving the data.

We'd like to be able to close indices to remove any memory overhead of them, but still have the indices available for replication.

Steps to accomplish this (not necessarily in order):

Routing Table

• Keep closed indices in the routing table and reinitialize shards when closing an index (Allow shards of closed indices to be replicated as regular shards #38024)
• Add the no op engine (from Replicate closed indices #31141) and plug it as the default engine for closed index shards ([RCI] Add NoOpEngine for closed indices #33903)
• Change the Gateway service to create routing table also for closed indices (Recover closed indices after a full cluster restart #39249)
• Adapt the Synced Flush Service to work when routing table exist for closed indices (SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence #37691)
• Ignore shard started requests when primary term does not match (Ignore shard started requests when primary term does not match #37899)
• Change IndexService so that it does not instantiates a mapper service/ index analyzers for closed indices (No mapper service and index caches for replicated closed indices #40423)

Open/Close Index APIs

• Close Index API should wait for closed shards to be started after an index is closed (Wait for shards to be active after closing indices #38854 )
• Before closing an index, the Close Index API should mark the index as read-only, wait for replication operations to be terminated, flush the index and then close it ([Close Index API] Refactor MetaDataIndexStateService #36354). It requires:
• Change IndexShardOperationPermits.asyncBlockOperations to use an action listener with a Releasable ([RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListener<Releasable>) #34902)
• Add methods to IndexShard that allow to acquire all index operations permits (Expose all permits acquisition in IndexShard and TransportReplicationAction #35540)
• Transport Replication Action must check blocks while having the index operation permit ([RCI] Check blocks while having index shard permit in TransportReplicationAction #35332, TransportResyncReplicationAction should not honour blocks #35795, Combine the execution of an exclusive replica operation with primary term update #36116)
• Add new type of TransportReplicationAction that can acquire all index operations permits during execution ([Close Index API] Add TransportShardCloseAction for pre-closing verifications #36249)
• Add unique id to cluster block ([Close Index API] Add unique UUID to ClusterBlock #36775)
• Release fresh listeners created by ?refresh=wait_for when close indices or relocating shards (Force Refresh Listeners when Acquiring all Operation Permits #36835)
• Close Index API should force a flush if a sync is needed (Close Index API should force a flush if a sync is needed #37961)
• Close Index API should issue a synced-flush so recoveries can be noop (Noop peer recoveries on closed index #41400)
• Remove index routing table of closed indices in mixed versions clusters (Remove index routing table of closed indices in mixed versions clusters #38955)
• What if the primaries are not all allocated? The verification step cannot go through or shards might not be properly flushed. Should we close the close index API? Should we allocate an empty routing table that cannot be allocated at all? (GatewayIndexStateIT.testRecoverBrokenIndexMetadata fails on master #40867) Perhaps add a "force" option to close index API so that by default the closing fails on unallocated primaries, but is ignored when explicitly forcing it. (We discussed this during GAH and agreed not to make this change. Users can reopen the closed index in this case).
• Improve Close Index and Freeze Index responses to report acknowledgments and shard failures per indices (and adapt High Level Rest client) (Improve Close Index Response #39687) (We still need to adopt this enhancement in HLRC).
• Flush shard at the end of recovery (so that newly recovered replica shard that are not yet in the in-sync allocations ids have 0 operations in translog if they are later opened as frozen) and add a test that closes an index under complex situations (e.g. network partitions) and checks that we can always allocate a "closed" or "frozen" primary. This is also the case with relocating shards (see Improve CloseWhileRelocatingShardsIT #37348 (comment)) (Peer recovery should flush at the end #41660)

Engine/Translog

• Add NoOpEngine as engine implementation to use for closed shards ([RCI] Add NoOpEngine for closed indices #33903)
• Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines(Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines #37426)
• Relax NoOpEngine constraints on translog (Relax NoOpEngine constraints on translog #37413)
• ReadOnlyEngine should update translog recovery state information (ReadOnlyEngine should update translog recovery state information #39238)
• Should we clean/trim the translog when an index is closed? (Compact index when closing #42445)
• Should we enforce a single index commit when an index is closed? (see comment) (Compact index when closing #42445)

Tests

• Add test for peer recovery of closed shard (Adapt the Recovery API for closed indices #38421)
• Add test for replica-to-primary promotion of closed shard (Add replica to primary promotion test for closed indices #39110)
• Add test for closed indices and full cluster restart (Recover closed indices after a full cluster restart #39249)
• Add test for closed indices and shard allocation filtering/awareness (Adapt more tests suites to closed indices #39186)
• Add test for closed indices and Cluster Reroute API (Adapt more tests suites to closed indices #39186) (Adapt the Recovery API for closed indices #38421)
• Test restoring a snapshot into a closed index
• Handle rolling upgrade from a version of ES that does not allocated closed indices to a version that does (maybe just an allocation decider that prevents the closed index shards to be assigned as long as all node are not on the current version?)
• Adapt IndexShardIT.testIndexCanChangeCustomDataPath() to closed indices (Adapt testIndexCanChangeCustomDataPath for replicated closed indices #38327)
• Adapt IndicesServicesTests.testPendingTasks() to closed indices (Adapt testPendingTasks() for replicated closed indices #38326)
• Test the Cluster Shard Allocation Explain API with closed indices (Test the Cluster Shard Allocation Explain API with closed indices #38631)
• Test that the "direct close" method still work in 7.x (removed in 8.0 with Remove index routing table of closed indices in mixed versions clusters #38955, see https://github.com/elastic/elasticsearch/pull/38955/files#r257718054)

Others

• Cluster Reroute API and force allocation of closed primary shard (Add test to verify force primary allocation on closed indices #42458)?
• Cluster Health API and closed indices (Adapt the Cluster Health and Cat Indices APIs to closed indices #39364)
• Recovery API and closed indices (Adapt the Recovery API for closed indices #38421)
• Indices Stats for closed indices will report 0 docs for closed indices/shards (Build DocStats from SegmentInfos in ReadOnlyEngine #34079 Return cached segments stats if include_unloaded_segments is true #39698)
• TransportUnfollowAction should increase settings version (TransportUnfollowAction should increase settings version #37859)
• Freezing an index should increase its index settings version (Freezing an index should increase its index settings version #37813)
• RestoreService should update primary terms when restoring shards of existing indices (RestoreService should update primary terms when restoring shards of existing indices #38177)
• Do not schedule refresh/translog fsync/translog trim/globalcheckpoint sync tasks for closed indices (Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices #38329)
• Investigate the behavior of retention leases sync task with replicated closed indices (see Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices #38329 (comment)) (won't do)
• Closing an index was sometimes used to change the custom data path setting (see Adapt testIndexCanChangeCustomDataPath for replicated closed indices #38327) but with replicated closed indices it means copying files of a live index. We can maybe extend the shrink/split API to do this instead or maybe just don't support changing the setting. (We discussed and agreed not to implement this feature for now given its complexity. We will revisit it in the future.)
• Investigate how to snapshot closed indices (Allow snapshotting replicated closed indices #39644)
• How to integrate closed replicated indices with closed follower indices (see Adapt CloseFollowerIndexIT for replicated closed indices #38767 (comment)) (Integrate closed replicated indices with closed follower indices #42442)
• GlobalCheckpointSyncAction triggers some warnings in logs after a replica to primary promotion. Should we just prevent maybeSyncGlobalCheckpoint() to be executed if the shard is closed? (Skip global checkpoint sync for closed indices #41874)

[elasticmachine]

Pinging @elastic/es-distributed