[Feature] Support Transform as an ISM action

[tanqiuliu]

Issue #, if available:
#733

Description of changes:

• Implemented ISMTransform, TransformAction model and parser
• Implemented two steps for TransformAction: AttemptCreateTransformJobStep & WaitForTransformCompletionStep
• Updated ActionProperties to include transform action execution context in the metadata
• Updated ism-config and ism-history mappings to include ism transform
• Modify ManagedIndexCoordinator to accept XContentRegistry from upstream instead of using empty registry, so that it can parse queries and aggregations.
• Unit tests & integration tests for transform action

Notes: The implementation and tests are mostly taking ISM Rollup as example.

Example policy with ISM transform:

{
    "policy": {
      "description": "hot transform delete workflow",
      "default_state": "hot",
      "schema_version": 1,
      "states": [
        {
          "name": "hot",
          "actions": [],
          "transitions": [
            {
              "state_name": "transform",
              "conditions": {
                "min_doc_count": "1440"
              }
            }
          ]
        },
        {
          "name": "transform",
          "actions": [
            {
              "transform": {
                "ism_transform": {
                    "description": "Creating transform through ISM",
                    "target_index": "target",
                    "page_size": 1000,
                    "data_selection_query": {
                      "match_all": {}
                    },
                    "groups": [
                      {
                        "date_histogram": {
                          "source_field": "timestamp",
                          "fixed_interval": "1h"
                        }
                      }
                    ],
                    "aggregations": {
                      "hourly_avg": {
                        "sum": {
                          "field": "value"
                        }
                      }
                    }
                  }
              }
            }
          ],
          "transitions": [
            {
              "state_name": "delete"
            }
          ]
        },
        {
          "name": "delete",
          "actions": [
            {
              "delete": {}
            }
          ]
        }
      ],
      "ism_template": {
        "index_patterns": ["test-data-ism-*"],
        "priority": 100
      }
    }
  }

CheckList:

• [ X ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Angie-Zhang]

@tanqiuliu Thanks for contribution. Could you check those failed workflows?

[tanqiuliu]

@Angie-Zhang The failures are all because of

Caused by: org.gradle.api.GradleException: Failed to create Jar file /Users/tanqiuliu/.gradle/caches/jars-9/94ac7eb5f0b504d7e3b47bc2eec907cc/jackson-core-2.15.0.jar.

This seems not related to this PR. Actually the same problem applies to main branch as well.

I've created another issue #763 for this problem and put my investigation there.

I upgraded to gradle 7.6.1 locally and no longer saw this exception. However, I ran into the the compileKotlin failure mentioned in #762 .

I guess I will wait for a few days before main branch is stablized.

[codecov]

Codecov Report

Merging #760 (399afba) into main (f543d93) will increase coverage by 0.14%.
The diff coverage is 80.61%.

@@             Coverage Diff              @@
##               main     #760      +/-   ##
============================================
+ Coverage     74.63%   74.78%   +0.14%     
- Complexity     2751     2794      +43     
============================================
  Files           361      366       +5     
  Lines         16077    16365     +288     
  Branches       2314     2346      +32     
============================================
+ Hits          11999    12238     +239     
- Misses         2805     2830      +25     
- Partials       1273     1297      +24     

Files	Coverage Δ
...pensearch/indexmanagement/IndexManagementPlugin.kt	89.45% <100.00%> (ø)
...anagement/indexstatemanagement/ISMActionsParser.kt	91.93% <100.00%> (+0.13%)	⬆️
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	63.15% <100.00%> (+0.12%)	⬆️
...exmanagement/opensearchapi/OpenSearchExtensions.kt	72.35% <100.00%> (-1.42%)	⬇️
...ent/indexstatemanagement/action/TransformAction.kt	70.00% <70.00%> (ø)
...dexstatemanagement/action/TransformActionParser.kt	57.14% <57.14%> (ø)
...ch/indexmanagement/transform/model/ISMTransform.kt	93.27% <93.27%> (ø)
...nt/step/transform/AttemptCreateTransformJobStep.kt	75.00% <75.00%> (ø)
...t/step/transform/WaitForTransformCompletionStep.kt	69.11% <69.11%> (ø)

... and 10 files with indirect coverage changes

[bowenlan-amzn]

@tanqiuliu finishes another round of review including the tests, very high quality changes!
Left only some minor comments. And we can merge this in after you got time to wrap up.

Thanks for setting up this high standard 👍

[bowenlan-amzn]

Since we don't support providing a scripted target_index name in transform, so one ISM policy with transform action would have fixed target_index, and all the managed indexes of this policy would transform into this one index.

We will support this probably in the future, using scripted target_index or support alias in target_index, relevant issue #656

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/index-management/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/index-management/backport-2.x
# Create a new branch
git switch --create backport/backport-760-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 3ddd75a85e3600cee7177c6f1d2acc4da660a161
# Push it to GitHub
git push --set-upstream origin backport/backport-760-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/index-management/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-760-to-2.x.

[aggarwalShivani]

Hi, is this new transforms feature documented? I could not find it under ISM supported actions at https://opensearch.org/docs/latest/im-plugin/ism/policies/#ism-supported-operations.

[sarthakaggarwal97]

@aggarwalShivani this is not backported to OS 2.x versions, and hence not released.

@bowenlan-amzn do we see any challenge in backporting this to 2.x?

[aggarwalShivani]

Hi @sarthakaggarwal97
I see this feature mentioned in release notes of version 2.12.0 released in February 2024.

[sarthakaggarwal97]

@aggarwalShivani thanks, looks like it was backported to 2.12 with #1002
@bowenlan-amzn checking if we followed this up with any documentation?

[bowenlan-amzn]

@sarthakaggarwal97 yes, we should add it like the rollup action here