[ILM] Allow ILM and CCR to work well together

[colings86]

If an index is a CCR leader or follower index then the delete and shrink action should wait proceding any operations. This is to avoid problems described under original problem description.

Leader indices

ILM needs to query the indices stats api and check the shard history retention leases in order to determine whether an index is a leader index.

If an index is a leader index then the delete and shrink actions first need to execute the following steps:

• Set the index.lifecycle.indexing_complete index setting to true.
• Periodically query the indices stats api and check whether there are no shard history retention leases for the leader index.

After this it is safe the proceed any steps that are part of the ILM delete and shrink actions.

Follower indices

ILM needs to check an index's custom index metadata to check whether an index is a follower index.
If an index is a follower index then the shrink action first needs to execute the following steps:

• Wait for the index.lifecycle.indexing_complete index setting to be replicated from the leader index.
• Then after that wait for the follower index's global check point to be equal to the leader index's global check point.
• Pause index following for the follower index. (This will release any shard history retention leases a follow index has on its leader index)
• Close the follower index.
• Unfollow the follower index. (Only closed indices can be unfollowed, because it changes the internal engine for all shards.)
• Open the unfollowed index.

After this it is safe the proceed any steps that are part of the ILM shrink action.

Tasks

• Change the delete and shrink actions to safely handle CCR leader indices. Ensure ILM policies run safely on leader indices  #38140
• Implement an Unfollow action [ILM] Add unfollow action #36970
• Inject Unfollow action/steps before Shrink and Rollover actions Inject Unfollow before Rollover and Shrink #37625

Original problem description

Currently if a user wishes to use CCR and ILM together on the same index they can run into problems. To help describe these problems imagine we have two clusters (for this discussion I'm going to call them leader and follower) and we are using CCR's auto-follow on the follower cluster to follow any indices on the leader cluster matching test-*.

Now, because in our scenario we have a time series use case it would also be good to have ILM manage the indices, so on the leader we set up a policy on the leader cluster which uses rollover, warm allocation, forcemerge, and shrink. Then we add the policy name to the index template for test-*, bootstrap ILM by creating the first index and now we have ILM working on our leader cluster and managing the test-* indices.

Problem 1 - Setting up a policy for the following indices

Having the test-* indices managed by ILM on the leader cluster is great but equally we would like ILM to manage the following indices on the follower cluster too. However, we can't use the exact same policy on the follower cluster because the following index will not have the write alias and even if it did we don't want the following index to rollover on its own criteria, we want it to mirror the leader index. This means the following index needs an indication that the leader has rolled over and moved to the warm phase so the following index also knows it can move to the warm phase.

Problem 2 - The leader index and the shrink action

In ILM the shrink action allocates one copy of each shard to a single node, then performs the shrink operation and then deletes the original (un-shrunk) index and sets an alias on the new (shrunken) index with the same name as the original index. This allows the naive user to search the index as if it was still the same index but under the covers the index is a different index.

The problem when combining this with CCR is that the following index may not be completely up to date with the leader index at the point the shrink action is performed, meaning that it may suddenly discover the leader index no longer exists and not be able to progress since there is no way for it to know that the index is equivalent to the shrunken index on the leader and means that the follower and leader cluster are indefinitely out of sync.

One solution to this would be for the un-shrunken following index to delete itself and for there to be a separate auto-follow rule to sync the shrunken indices from the leader. The problem with this is that it requires all the follower shrunken index to be synced from scratch copying all the same data as it had already in the un-shrunken index which is a waste of resources but more importantly means there is a period where the follower cluster will actually be getting further out of sync with the leader since its thrown away the un-shrunken index and is waiting to fully sync the shrunken index from the leader.

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

@gwbrown and I started to think about a high level plan for how to make ILM work with CCR. It is quite abstract since it is unknown yet how certain primitives will be implemented in CCR.

The high level plan is built on the assumption that ILM is currently targeted at time-series use cases,
in which there is a period where documents are indexed (the write period), followed by a period where no documents are indexed or updated ever again (the read-only period). The other assumption is that each cluster has own ILM policies. A leader cluster shouldn't propagate actions / steps to follower cluster and visa versa.

CCR will add a mechanism to indicate in the leader cluster that a leader index is being followed by one or more indices from another cluster. A follower cluster should indicate this prior to following a leader index in the leader cluster.

CCR will also add a mechanism for the ILM to specify that a leader index is “done indexing” and will be read-only thereafter - that is, it has left the write period and entered the read-only period. The fact that a leader index is read only is also replicated to follower index. Once a leader index has been marked as “done indexing”, follower clusters will continue following until they have replicated all updates, then automatically unfollows (1). CCR in the following cluster will then indicate to the leader cluster that this follower index no longer follows the leader index.

On the leader cluster:

• ILM will be able to use these mechanism to mark an index as “done indexing” once the index has been rolled over.
• ILM will wait execute actions / steps that are destructive for CCR leader indices. These ILM steps wait until the indication that an index has followers has been removed. (0 indices are following a leader index)

On the follower cluster:

• ILM will not execute actions / steps that are destructive for CCR on follower indices. These ILM actions / steps will wait until follower indices have been unfollowed and the ready only attribute has been set.

1: This would be an operation that: pauses index following, closes, unfollows and then opens the index.

[martijnvg]

We discussed yesterday in more detail what the read only attribute is, in order to let ILM safely operate describe operations in the follower side.

ILM can use the readonly action to set the index.blocks.write index setting. If that index is being followed then CCR will automatically replicate that to the corresponding follower index in a follower cluster. ILM in a following cluster can then automatically unfollow a follower index if index.blocks.write index setting has been set and the global checkpoint of a shards in a follow index are equal to the global checkpoint in the corresponding leader index. When the follower index has been unfollowed then ILM is allowed to execute destructive operations on this index (shrink or delete action), since it has become a regular index. ILM should not perform destructive operations on follower indices (this can be checked by checking custom metadata in IndexMetaData).

This part of the CCR ILM integration can already be implemented, so we should start soon.

[colings86]

We discussed using the index.blocks.write index setting before and said that its not a good idea to use this setting and we should have a different setting for this. This is because users may set the write block themselves for other reasons and then unset it later intending to keep indexing into that index. Therefore we cannot be sure this setting means the indexing is complete for that index. I think we should do this on a separate setting. I opened #35944 this morning which is basically the same mechanism but using a different setting which would solve this part of the CCR ILM issue and also be good for other use cases too. We can also make sure that once this new setting is set it cannot be unset which I think is important for these cases

[colings86]

ILM should not perform destructive operations on follower indices (this can be checked by checking custom metadata in IndexMetaData).

Is this saying that follower indexes should never run shrink or delete? If so I don't think this restriction is necessary since the index will no longer be following by the time it gets to the warm phase

EDIT: actually I see what this means now. We should not perform destructive operations if the follower index is still following. In this case we should go to the ERROR step. In the case that things have worked as expected the index would not be a follower index anymore but a regular index so would be fine to run destructive operations

[martijnvg]

This is because users may set the write block themselves for other reasons and then unset it later intending to keep indexing into that index.

This was discussed yesterday and the conclusion was that this is problem also in other scenarios. Something like a readonly api should be build that turns an index into a read only index forever. But that should be tackled outside ccr / ilm.

Is this saying that follower indexes should never run shrink or delete?

Well not when it is actively following a leader index. The index would first need to be unfollowed before that could be done.

[martijnvg]

actually I see what this means now. We should not perform destructive operations if the follower index is still following. In this case we should go to the ERROR step. In the case that things have worked as expected the index would not be a follower index anymore but a regular index so would be fine to run destructive operations

👍

[colings86]

This was discussed yesterday and the conclusion was that this is problem also in other scenarios. Something like a readonly api should be build that turns an index into a read only index forever. But that should be tackled outside ccr / ilm.

I agree that we should build a readonly API that makes the index readonly forever but I don't think using the write block is a good idea in the interim. Users may already be used to using the write block as part of maintenance workflows, if the index reacts to that setting at a time that the user does not expect then the results are pretty bad since the index will un-follow the leader when its up to date meaning it is not going to be a copy of the leader. I think we should use something other than the write block.

[martijnvg]

I see, that makes sense. So instead of relying on index.blocks.write, ILM on a follower cluster would rely on something like index.lifecycle.index_complete. Should this new index setting also control whether a write block is set? Because otherwise documents can still be indexed while this setting is set.

[colings86]

Should this new index setting also control whether a write block is set? Because otherwise documents can still be indexed while this setting is set.

This is definitely up for discussion but I think we need something that has a clear intention and meaning that the index is not intended to be written to agian