inherit [index.lifecycle.date] from rolled-over time

[talevy]

Indices that are rolled over preserve their origin index.lifecycle.date
as the time the index was created. Although this is also fine, it would
make more sense to start the timer for moving to the warm phase from the
time it was rolled over so that we capture the time elapsed from the
latest data, as opposed to when the index was created.

This commit adds an extra step within the RolloverAction that
extracts the index.creation_date of the newly created index from the RolloverStep
and sets that as the index.lifecycle.date of the rolled-over index that is
waiting for its next phase.

[elasticmachine]

Pinging @elastic/es-core-infra

[colings86]

I can see a potential issue with this approach. There is no guarantee that the set will run while the next index has the alias. In normal execution this is technically true but unlikely but it becomes more likely if the user sets index.lifecycle.skip on the index after the rollover step completes but before this step runs. In that case there is a scenario where the rollover of the next index happens before the index is un-skipped and it will write the wrong timestamp to the index. We need to make sure that the timestamp is always correct since if we add the wrong timestamp it affects the index for the entire of its lifetime.

I wonder if we should instead modifying the TransportRolloverAction so as well as changing the alias to the new index it also writes a setting to the old index to indicate when it was rolled-over? This seems like a generally useful piece of information to the user even outside the scope of ILM. That way we can have this step work much like the InitializePolicyContextStep where it just takes the timestamp from that new setting and sets it on the index.lifecycle.date setting.

[talevy]

I've opened #30887 to discuss

[talevy]

Hi @colings86, I've updated this PR to leverage RolloverInfo

[colings86]

I'm not sure if we should have this as an async action step or have it work directly on the cluster state update thread like PolicyInitialisationStep does? It seems strange to me that we set this setting on the cluster state update thread initially and then later as an async action step.

[talevy]

good catch, it was a reflex for updating settings... but this setting is controlled by us and safe to do on the cluster-state 👍

[colings86]

Left a minor comment bbut LGTM

[colings86]

As this step is special and we rely on it definitely running as is I think wee should make it final.

Actually I think we should make all the steps final but that can be in a different change. I think its most important that this one is final as we need to ensure the creation date is definitely set properly