Allow a dynamic target index in ISM Rollup

[adityaj1107]

Issue by mark-meyer
Wednesday Mar 31, 2021 at 00:46 GMT
Originally opened as opendistro-for-elasticsearch/index-management#428

---

I would like to apply a policy to daily log indices that performs a rollup in one of the states before deleting the index. However is seem the target_index requires a string and there is no option to interpolate something like {{ctx.index}} or similar. This makes it impossible to apply this policy to more than one index unless you are okay with rollup overwriting the previous one, which you probably aren't.

Describe the solution you'd like
I would like to specify the target_index field of an ism rollup dynamically based on the name (or some other metadata) of the index being rolled up such as "target_index": "rollup_{{ctx.index}}"

[adityaj1107]

Comment by dbbaughe
Wednesday Mar 31, 2021 at 00:51 GMT

---

I believe @thalurur just did something similar for snapshot name too.

That being said, correct me if I'm wrong Ravi, but if @mark-meyer applies this policy to daily log indices even if they roll up into the same target index they will not overwrite each other. We do support multiple rollups in the same index. Each index that is managed by the policy would have a rollup job created for itself when the rollup action is created which will use its own rollup ID as a namespace in the target index to handle conflicts. And the rollup ID seems to consist of a hash of the job configuration and the source index which will be unique per daily log index.

Edit: Do we delete these temporary rollup jobs after they finish @thalurur?

[adityaj1107]

Comment by thalurur
Monday Apr 05, 2021 at 22:30 GMT

---

At the moment ISM action will not delete the jobs after they are finished. Rest of the behavior described by Drew is accurate - at the moment the same policy will be creating multiple jobs (based on the source index the policy is being executed on) and all of these jobs writing data to the same target index. And data from each job cannot be touched by other jobs as their namespace includes the rollup job id.

Rollup implementation have some constraints imposed at the moment:

1. If multiple rollup jobs have written data to same target index then data from only one job is picked when searching the rollup index, there is no ability to make the search check through all the jobs at the moment.
2. If search requests consists of multiple rollup indices then the search on rollup indices is failed.

The above makes the only way to rollup data for daily log indices is to create one job that writes to one target index. So the rolled up data can be searched together.

What happens with current policy:
In this case for each new daily log index a new rollup job is created, but all the jobs write to the same target index.
Though the documents from different rollup jobs inside target index are shielded from overwriting, fetching documents from this index is constrained by constraint "1". During search on the target index a single job is picked from multiple jobs and this will prevent ever fetching all the data or user desired job data from target index.

What happens with proposed policy:
If we update the rollup ISM policy to take in a scripted target index, then for each new daily log index a new rollup job is created, writing to a new target index.
In this case since only one job is writing to target index during search there is no issue of which job is created and user can pick which index they want to search from. But, if user want to search across target indices or all target indices its not possible.

We have few options if we need to support either:

• Adjust the constraint "2" - multiple rollup indices cannot be searched together. Adjusting this constraint and making the ISM rollup policy target index scriptable like requested will make it possible to search across all ISM policy created rollup jobs as long as all jobs wrote to different index.
• Adjust the constraint "1" - make it possible to search multiple rollup job data on the same target index, this makes the current policy work
• A combination of both - make it possible to search multiple rollup jobs on same target index and allow multiple rollup indices to be searched in same search request

@mark-meyer Couple of questions:

1. Is there an use case where you would need to search all the data from different source indices that is rolled up through ISM policy together? (I am assuming you do cause since rollup definition is same in policy for all jobs, the data is just spread out temporally)
2. If 1 is yes, do you need the data to be rolled up to different indices or having it single index works as well?

[adityaj1107]

Comment by garlicsauce
Wednesday Apr 28, 2021 at 11:00 GMT

---

+1

I was trying to setup rollup policy with target index containing date math expression "target_index": "<test-{now/d}>",. Index was created - test-2021.04.28 however rollup job fails with an error saying

Status
Failed: Failed to update mappings of target index [<test-{now/d}>] with rollup job

Without such possibility I guess I need to create some script running every day that would create rollup jobs itself which is much uglier solution. To sum up - it would be nice to be able to setup a rollup job/rollup policy with dynamic target index.

[dbbaughe]

Additional comments:

• Be able to use an alias as the rollover target
• Be able to have date math in the rollover target index name so it dynamically changes each day
• Allow for dynamic index naming based off index rolled up (e.g., if done via ISM action on a rolled index, against a source index of logs-90-000422, a config like %{indexname}-rollup would create logs-90-000422-rollup)

Main issue being that the target of the rollup is currently static, which means once that target index is too large then you need to manually stop the rollup job and start a new one to point to a new index. We need to be able to have a single continuous rollup job that can write to a target that can change (by using an alias where backing indices are rolled over, maybe data streams, dynamically configured index names, etc).

[setiah]

To sum up, the ask is

1. Allow target-index name in a rollup job to be dynamic so it is possible to rollup data into multiple time-based indices instead of one index (as it is today).
2. As a follow up to 1, when the data is available in multiple target indices, a user should be able to run search across all rolled up indices. Currently search is only supported for one rollup index at a time since the rollup data goes into just one target index. Tracking issue - Allow rollup searches to multiple rollup indices if they have the same rollup #321

@mark-meyer @garlicsauce does this address your use case?

[petardz]

@setiah

Few question regarding your first comment:

1. When creating rollup job, are we going to store resolved target_index value?
2. Do we need to support anything other then "target_index": "rollup_{{ctx.source_index}}", like date math expressions?
3. if answer to question 1 is no,(we're resolving it during job run) what are we returning in GetRollup(s)Action: resolved value or scripted?