[ILM] Avoid race condition between shrinking and ILM itself

[nielsbauman]

In the ILM Shrink action, ILM performs (among others) the following steps:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/ShrinkAction.java

Lines 174 to 176 in 0ae817e

	StepKey shrinkKey = new StepKey(phase, NAME, ShrinkStep.NAME);
	StepKey enoughShardsKey = new StepKey(phase, NAME, ShrunkShardsAllocatedStep.NAME);
	StepKey copyMetadataKey = new StepKey(phase, NAME, CopyExecutionStateStep.NAME);

The ShrinkStep copies the ILM policy name in the shrink request:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/ShrinkStep.java

Line 85 in 0ae817e

.put(LifecycleSettings.LIFECYCLE_NAME, policyName)

However, when the allocation of the shards of the shrunken index takes "some" time (which is not unexpected for larger shards), which is what the second step waits for, ILM starts executing the policy on the shrunken index before the ILM execution state has been copied to that index (step three). This means ILM will start executing the policy from the beginning for the shrunken index, causing the WaitForRolloverReadyStep to fail here, instead of proceeding in the shrink action.

This is not necessarily harmful, it's just "noisy" - there is no data loss, it only produces some error logs and possibly a yellow health API status.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

After some internal discussions, we came up with the solution to (re)add an index setting that tells ILM to skip the index. This setting should be added in the ResizeRequest and removed when we add the execution state to the shrunken index (i.e. after the resize request/shard allocation has completed). See #34823 for inspiration - that PR removed such a setting due to a lack of use cases.