Fixed index resolution in transform jobs to correctly resolve the backing indices of a data stream

[ketanv3]

Description

Fixed the index expression resolution to correctly resolve the backing indices of a data stream during the validation of transform jobs.

To create the transformed results, the underlying search request uses org.opensearch.action.search.SearchRequest which already works with data streams as expected. So no other change is expected.

Issues

• Closes [BUG] Transform jobs do not resolve the backing indices of a data stream #71

Check list

• Commits are signed per the DCO using --signoff
• All tests pass
• New functionality includes tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma ketan9495@gmail.com

[thalurur]

Can you introduce a test for this?

posted a comment

[ketanv3]

@thalurur Added an integration test for this.

Also, to provide more context about the large diff in src/test/resources/data/nyc_5000.ndjson – I have updated the op_type for bulk requests from index to create. This is done because a data stream only allows create.

This should not impact the execution of other tests which use this sample data. This is because document IDs are auto-generated (thus, unique) and there's no requirement for updating existing documents.

[codecov-commenter]

Codecov Report

Merging #72 (bc0e029) into main (9ec5d91) will increase coverage by 0.51%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main      #72      +/-   ##
============================================
+ Coverage     78.08%   78.60%   +0.51%     
- Complexity     1858     1873      +15     
============================================
  Files           255      255              
  Lines         10267    10267              
  Branches       1518     1518              
============================================
+ Hits           8017     8070      +53     
+ Misses         1430     1360      -70     
- Partials        820      837      +17     

Impacted Files	Coverage Δ
...ch/indexmanagement/transform/TransformValidator.kt	59.18% <100.00%> (ø)
...ensearch/indexmanagement/rollup/model/ISMRollup.kt	93.04% <0.00%> (-1.74%)	⬇️
.../rollup/action/start/TransportStartRollupAction.kt	68.65% <0.00%> (-1.50%)	⬇️
...nt/rollup/action/stop/TransportStopRollupAction.kt	75.75% <0.00%> (ø)
...earch/indexmanagement/transform/model/Transform.kt	90.15% <0.00%> (+1.03%)	⬆️
...ent/indexstatemanagement/util/ManagedIndexUtils.kt	78.69% <0.00%> (+1.30%)	⬆️
...dexmanagement/transform/model/TransformMetadata.kt	90.36% <0.00%> (+3.61%)	⬆️
...indexmanagement/rollup/model/RollupFieldMapping.kt	83.33% <0.00%> (+8.33%)	⬆️
...xmanagement/rollup/actionfilter/FieldCapsFilter.kt	73.52% <0.00%> (+27.64%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ec5d91...bc0e029. Read the comment docs.