[EPM] Reduce number of indices on upgrade

[ruflin]

The currently documented upgrade path for packages should work but has the side effect of potentially creating many indices. Assuming a package has 10 datasets which are used by the user and is used in 2 namespaces, this leads to 20 indices. Now every time the package is updated, 20 more indices are added as the rollover is triggered for the new mappings and ingest pipeline to apply.

Most package updates are backward compatible and only a subset of the datasets is updated. This allows to reduce the number of newly created indices with the following proposal. Kibana assets are skipped below.

• Package is updated
• New ingest pipeline is loaded into ES
• New Elasticsearch Alias template is loaded
• Update mappings of all dataset indices with POST {alias}/_mapping
• Update the pipeline settings of all dataset indices with POST {alias}/_settings.
• In case of an error during update of a dataset, trigger rollover for this dataset
• Remove old ingest pipeline

The above will have the effect, that only indices are rolled over where a conflict with the previous mapping exist. These then directly get rolled over and the new index template is applied. In case a rollover is triggered by ILM in the middle of the above process, it is also not a problem as the updated Elasticsearch template will already exist.

[elasticmachine]

Pinging @elastic/ingest (Feature:EPM)

[skh]

To check my understanding:

New Elasticsearch Alias template is loaded

• At this point we have both the old and the new alias template?
• This means alias templates need to include the version in their name?
• Will old alias templates ever be needed to cleaned up?

Update mappings of all dataset indices with POST {alias}/_mapping
Update the pipeline settings of all dataset indices with POST {alias}/_settings.

• These update mappings and settings for only the index {alias} is currently pointing at for writing, correct?

In case of an error during update of a dataset, trigger rollover for this dataset

How is a rollover triggered manually / programatically? As described in https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html , just with a condition that is guaranteed to be true?

[ruflin]

• We will only have the new alias template, the old one gets overwritten. No version umber needed. Is there a reason we would need to keep the old template around? If we overwrite it, we also don't need to clean it up.
• Pointing to alias for writing: Yes. It is possible to send the update requests to the Alias and ES will do the right thing.
• Triggering rollover: Good question about the force rollover. In my tests I used max_docs: 1 but I would prefer if there is an option to force a rollover. Perhaps the ES team has an idea on this?

[skh]

Is there a reason we would need to keep the old template around?

No, I misunderstood the original description, thanks for the clarification.

[ruflin]

@neptunian As you are working on the upgrades at the moment, this shares some background on why we do it. Since filing this a few things have changed, so I wonder if we can close this issue in favor of your upgrade issue?

[neptunian]

Yes, I will reference it.