Improve integration with data streams

[ketanv3]

Description

This PR aims to resolve the limitations/enhancements pointed out in #12. Here's a short summary.

1. Fixes the ISM policy association behaviour when a new backing index of a data stream is created.
2. When executing the rollover action on a backing index of a data stream, ISM will automatically identify the data stream name and use it as the rollover target. Users no longer need to specify the rollover_alias index setting.
3. Fixes the index expression resolution in rollups when the source is a data stream. The data stream name will be resolved to its corresponding backing indices.

Issues

• Closes [Feature] Improve integration with data streams #12
• Relates [Feature] Support Data Streams in OpenSearch OpenSearch#675

CheckList

• Commits are signed per the DCO using --signoff
• New functionality includes tests
• All tests pass

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma ketan9495@gmail.com

[ketanv3]

Please note that the newly added integration tests are expected to fail.

Changes to add data streams' support in OpenSearch have been merged recently (opensearch-project/OpenSearch#690), but the latest artifacts are yet to be made available.

The test clusters launched from the existing artifact (opensearch-1.0.0-beta1-linux-x64.tar.gz) do not support the creation of a data stream, thus, newly added integration tests that involve the creation of a data stream are expected to fail. Will have to re-run the checks once the latest artifacts are available.

[codecov-commenter]

Codecov Report

Merging #13 (ec79512) into main (72018d4) will decrease coverage by 0.04%.
The diff coverage is 73.52%.

@@             Coverage Diff              @@
##               main      #13      +/-   ##
============================================
- Coverage     77.31%   77.27%   -0.05%     
- Complexity     1557     1561       +4     
============================================
  Files           201      201              
  Lines          8588     8603      +15     
  Branches       1326     1331       +5     
============================================
+ Hits           6640     6648       +8     
- Misses         1250     1256       +6     
- Partials        698      699       +1     

Impacted Files	Coverage Δ	Complexity Δ
...xmanagement/rollup/actionfilter/FieldCapsFilter.kt	45.39% <16.66%> (-1.15%)	21.00 <0.00> (ø)
...atemanagement/step/rollover/AttemptRolloverStep.kt	56.09% <76.47%> (+0.44%)	16.00 <6.00> (+2.00)
...agement/indexstatemanagement/ISMTemplateService.kt	78.72% <100.00%> (+2.53%)	0.00 <0.00> (ø)
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	74.45% <100.00%> (ø)	47.00 <0.00> (ø)
...arch/indexmanagement/rollup/RollupMapperService.kt	58.87% <100.00%> (ø)	23.00 <0.00> (ø)
...management/rollup/interceptor/RollupInterceptor.kt	68.42% <100.00%> (-0.55%)	31.00 <0.00> (ø)
...nt/indexstatemanagement/model/destination/Chime.kt	40.90% <0.00%> (-13.64%)	2.00% <0.00%> (-2.00%)
...anagement/indexstatemanagement/model/Transition.kt	64.61% <0.00%> (-3.08%)	5.00% <0.00%> (ø%)
...ensearch/indexmanagement/rollup/model/ISMRollup.kt	94.78% <0.00%> (+1.73%)	29.00% <0.00%> (+2.00%)
...nt/indexstatemanagement/model/destination/Slack.kt	54.54% <0.00%> (+13.63%)	4.00% <0.00%> (+2.00%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72018d4...ec79512. Read the comment docs.

[bowenlan-amzn]

Curious about index_patterns format here. Take logs-redis for an example, normally we will use logs-redis* as the index pattern with * in the end. But it seems here we use logs-redis, is this sth by design? And logs-redis* is wrong?

[ketanv3]

The index pattern matches the name of the data stream instead of the backing indices. This is by design and consistent with how index patterns work in index templates as well.

A data stream named logs-redis will have backing indices like .ds-logs-redis-000001, .ds-logs-redis-000002 and so on. When defining an index pattern like index_pattern: ["logs-redis"], I'm just performing an exact match on one data stream.

I could also define index_pattern: ["logs-redis-*"] which would allow me to match multiple data streams (and effectively their backing indices). This would match "logs-redis-prod", "logs-redis-staging" and so on.

[bowenlan-amzn]

Pls help update our release note for rc1 before merge in: https://github.com/opensearch-project/index-management/blob/main/release-notes/opensearch-index-management.release-notes-1.0.0.0-rc1.md

Add this part to the release note

### Enhancements

* Improve integration with data streams ([#13](https://github.com/opensearch-project/index-management/pull/13))

[bowenlan-amzn]

We have looked into the one failure in multi-node tests. test change policy API should only apply to indices in the state filter

This is a flakey one, doesn't seem to be related to changes in this PR.

After adding some log, we see the failed reason is first_index is still transitioning when we are making the change policy API call. But this cannot happen if the test is being executed sequentially. So it seems multi-node test can somehow execute the test somewhat parrelly. Will need to look into this in the future.