index going to "completed" ILM phase instead of "warm"

[DanRoscigno]

Bug report

Elasticsearch version
7.2.0 and 7.3.1 ESS

Description of the problem including expected versus actual behavior:
When I look at my indices under index management and apply the Lifecycle filters (hot, warm, cold) I see that I have a bunch of Filebeat indices that are not in any of the above states. When I look at the details I see the state is "completed":

Using the following ILM policy index should go to "warm" on rollover:

{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_age": "1h",
                        "max_size": "50mb",
                        "max_docs": 100
                    }
                }
            },
            "warm": {
                "min_age": "1h",
                "actions": {
                    "allocate": {
                        "include": {},
                        "exclude": {},
                        "require": {
                            "data": "warm"
                        }
                    }
                }
            }
        }
    }
}

Steps to reproduce:

1. Run filebeat setup
2. Edit ILM policy to reduce time (so you do not have to wait 30 days)
3. Run filebeat
4. Wait for rollover

[DanRoscigno]

@TomLawler : Change the filter on your indices from ilm.phase:(warm) to ilm.phase:(completed) and see if they show up.

[elasticmachine]

Pinging @elastic/es-core-features

[dakrone]

@DanRoscigno after the rollover, ILM moves the index into the warm phase and then immediately executes the allocate action, then moves the index into the complete phase. It sounds like you're expecting it to stay in the warm phase forever? This is expected behavior from ILM.

[DanRoscigno]

Thanks @dakrone ,
In my case (where I have no cold phase) does complete mean that the index is warm because it is available for searching and on a warm node and also complete because I have no further phases to go through?

If this is the thinking, then I would like it to show up in the ILM UI when I set the filter to show me warm indices:

From looking at the way the UI works, setting the phase to warm-completed would work (the filter matches something like *<choice>*

[dakrone]

In my case (where I have no cold phase) does complete mean that the index is warm because it is available for searching and on a warm node and also complete because I have no further phases to go through?

That's right, ILM is single pass execution, and once a policy has been completely executed it moves to the complete phase, so in this case, it executed the actions in the warm phase and then moved to complete because there is nothing left to do in warm.

If this is the thinking, then I would like it to show up in the ILM UI when I set the filter to show me warm indices:

This is something that would probably have to be addressed on the UI side, perhaps @bmcconaghy who worked on the original UI could direct you to where to file an appropriate issue for it?

[bmcconaghy]

Hmm this seems like a deficiency in the ES API. Not sure how the UI could figure out that something is still in warm phase if the phase is "complete". We filter based on phase (just a dumb string filter).

[dakrone]

I think the disconnect is thinking that an index resides in a phase permanently, for example, an index is only in the "warm" phase whilst operations are being executed in that phase.

To clarify this, we could potentially add a "last_phase" output to the ILM explain output for an index so the UI could use phase = phase == "completed" ? : last_phase : phase as the filtering criteria?

[bmcconaghy]

To be clear, there isn't really any "logic" to the filtering, it's a dumb UI component. I'm sure something could be coded to make this filtering work for the user, but the data needs to be in the return from ES. An issue should be filed in the Kibana repo linked to this one for the corresponding Kibana work.

[dakrone]

Another thing to discuss, is this something we event want? Perhaps a completed option should be added to the UI to show indices in the completed phase? Is there a benefit from trying to keep an index "in a phase" it's not currently in for the UI?

[bmcconaghy]

To me, "completed" is not a phase. An index stays in the last defined phase, so in the case described in this issue, it remains in "warm". This seems like what the person who filed this issue thinks makes the most sense, and I agree. "Completed" is about the lifecycle as a whole, not a phase I think.

[bmcconaghy]

I would defer to Elasticsearch UI on this one, though, as I don't work on this any longer. @cjcenizal any thoughts on this one?

[gwbrown]

My concern with showing complete indices as the last phase they were in before completion in the UI is that it might lead to users thinking that, e.g. "Since this policy only had hot and warm phases, I can add a cold to it and all my indices will immediately go into the new cold phase!" (which will not work, at least at the moment). It's already confusing for some users that editing a phase in a policy won't retroactively apply to indices which have already been through the changed phase, and I don't want to add on to that.

[DanRoscigno]

@dakrone : Do we need completed at all? Does this phase prevent some actions from considering those indices for action? If there is a check for phase == completed, maybe the checks could look at current action instead? Notice that both phase and current action here are completed: