[Rule based auto-tagging] Add autotagging rule integration tests #18550

Lindsay-00 · 2025-06-18T08:49:32Z

Description

Add Integration Tests for workload management rule based autotagging.

This IT can be run using
./gradlew :plugins:workload-management:internalClusterTest --tests "org.opensearch.plugin.wlm.WlmAutoTaggingIT"

Tests include:

testExactIndexMatchTriggersTagging
Verifies that a rule with an exact index match correctly triggers tagging when a search query is run on that index.
testWildcardBasedAttributesAreTagged
Ensures that rules using wildcard index patterns (e.g., logs-*) successfully apply tagging when matching queries are executed.
testMultipleRulesDoNotInterfereWithEachOther
Confirms isolation between multiple auto-tagging rules and workload groups, ensuring that only the matching rule's completions increase.
testTaggingTriggeredAfterRuleUpdate
Verifies that tagging is correctly triggered after updating an existing rule to match a target index (transition from non-matching to matching).
Nonexistent Workload Group ID
Validates that creating a rule with a nonexistent workload group ID throws a validation exception.
Too Many Index Patterns
Ensures that a rule is rejected if more than 10 index patterns are provided for an attribute.
Empty Index Pattern Value
Confirms that an index pattern with an empty string is invalid.
Index Pattern Exceeding Max Length
Checks that index pattern values longer than 100 characters are not allowed.
Delete Rule with Nonexistent Rule ID
Verifies that trying to delete a rule that doesn’t exist fails with an appropriate error.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-06-18T10:08:06Z

✅ Gradle check result for d4b9e08: SUCCESS

codecov · 2025-06-18T10:08:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.91%. Comparing base (6c4f3d8) to head (fbed93b).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18550      +/-   ##
============================================
+ Coverage     72.88%   72.91%   +0.03%     
+ Complexity    69841    69824      -17     
============================================
  Files          5673     5673              
  Lines        320756   320754       -2     
  Branches      46370    46367       -3     
============================================
+ Hits         233796   233893      +97     
+ Misses        68102    67918     -184     
- Partials      18858    18943      +85

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ruai0511 · 2025-06-19T00:12:29Z

Thanks @Lindsay-00 for the tests! For this feature we need integration tests (IT) to verify the end-to-end behavior across the system. However, the current code seems more on the UT side.

Quick recap: Unit Test (UT) vs. Integration Test (IT)
Unit tests (like the current ones using OpenSearchTestCase) focus on isolated components in-memory — no actual cluster, index, or API is involved.
Integration tests (ITs) spin up a test OpenSearch cluster and verify real-world behavior for a feature from end to end. It's like writing down the steps for the Functional testing you finished.

That being said, the current test class is extending a unit test base class (OpenSearchTestCase), but for ITs it should extend classes like OpenSearchIntegTestCase so that it spins up a test cluster and exercises real components.

Ideally each test should include

Creating a workload group (you can reuse the helper method from WorkloadManagementIT)
Creating an index ingest rules into it
Issuing search queries using eg. client().prepareSearch(...) and observing whether they get tagged with the correct workload group

Let me know if you have any questions. Thanks!

github-actions · 2025-06-21T01:11:55Z

✅ Gradle check result for 3952431: SUCCESS

github-actions · 2025-06-21T02:57:00Z

❌ Gradle check result for 99d3554: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-06-25T22:58:56Z

✅ Gradle check result for fceb87a: SUCCESS

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-06-27T22:16:20Z

❌ Gradle check result for 17a3568: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-06-30T08:16:02Z

❌ Gradle check result for f331044: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-06-30T08:33:13Z

❌ Gradle check result for 650ea46: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ruai0511

Could we add tests to verify the actual 'tagging' part? As mentioned before, IT is end-to-end testing, so we should have tests that include the complete flow: from creating the rule, to running the query, to verifying whether the query has been tagged with the group ID.

...load-management/src/internalClusterTest/java/org/opensearch/plugin/wlm/WlmAutoTaggingIT.java

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-07-01T01:51:39Z

❌ Gradle check result for f09deac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-09-17T22:11:56Z

❌ Gradle check result for b46987f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

github-actions · 2025-09-17T23:25:37Z

❌ Gradle check result for 5f5ee47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk

LGTM!

jainankitk · 2025-09-18T01:41:37Z

Gradle failure looks unrelated:

[org.opensearch.gateway.remote.RemoteClusterStateServiceTests.testReadClusterStateInParallel_ExceptionDuringRead](https://build.ci.opensearch.org/job/gradle-check/63819/testReport/junit/org.opensearch.gateway.remote/RemoteClusterStateServiceTests/testReadClusterStateInParallel_ExceptionDuringRead/)

have retried the gradle check

github-actions · 2025-09-18T02:25:24Z

❌ Gradle check result for 5f5ee47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk · 2025-09-18T06:06:33Z

Same test still failing:

[Test Result](https://build.ci.opensearch.org/job/gradle-check/63832/testReport/) (1 failure / +1)

    [org.opensearch.gateway.remote.RemoteClusterStateServiceTests.testReadClusterStateInParallel_ExceptionDuringRead](https://build.ci.opensearch.org/job/gradle-check/63832/testReport/junit/org.opensearch.gateway.remote/RemoteClusterStateServiceTests/testReadClusterStateInParallel_ExceptionDuringRead/)

@Lindsay-00 - can you try merging from main?

github-actions · 2025-09-18T07:59:33Z

✅ Gradle check result for fbed93b: SUCCESS

…nsearch-project#18550) --------- Signed-off-by: Lingxi Chen <lingxich@amazon.com> Signed-off-by: Lindsay-00 <54655271+Lindsay-00@users.noreply.github.com> Co-authored-by: Lingxi Chen <lingxich@amazon.com>

…nsearch-project#18550) --------- Signed-off-by: Lingxi Chen <lingxich@amazon.com> Signed-off-by: Lindsay-00 <54655271+Lindsay-00@users.noreply.github.com> Co-authored-by: Lingxi Chen <lingxich@amazon.com> Signed-off-by: Ankit Jain <jainankitk@apache.org>

…nsearch-project#18550) --------- Signed-off-by: Lingxi Chen <lingxich@amazon.com> Signed-off-by: Lindsay-00 <54655271+Lindsay-00@users.noreply.github.com> Co-authored-by: Lingxi Chen <lingxich@amazon.com>

cwperks · 2025-09-25T18:35:33Z

...load-management/src/internalClusterTest/java/org/opensearch/plugin/wlm/WlmAutoTaggingIT.java

+        createRule(ruleId, "initial non-matching rule", "random", featureType, workloadGroupId);
+
+        // Wait for rule to be available
+        Thread.sleep(5500);


Can we please try to avoid using Thread.sleep at all costs? Wait for a condition to be met and proceed otherwise fail the test after some timeout if the condition is not met.

…sts (opensearch-project#18550)" This reverts commit 4b9aa27.

…sts (opensearch-project#18550)" This reverts commit 4b9aa27. Signed-off-by: Andrew Ross <andrross@amazon.com>

…nsearch-project#18550) --------- Signed-off-by: Lingxi Chen <lingxich@amazon.com> Signed-off-by: Lindsay-00 <54655271+Lindsay-00@users.noreply.github.com> Co-authored-by: Lingxi Chen <lingxich@amazon.com>

Lindsay-00 requested a review from a team as a code owner June 18, 2025 08:49

Lindsay-00 force-pushed the rule_IT branch from 00607c7 to 91fe58b Compare June 18, 2025 08:50

Lindsay-00 changed the title ~~Add autotagging rule integration tests~~ [Rule based auto-tagging] Add autotagging rule integration tests Jun 18, 2025

Lindsay-00 force-pushed the rule_IT branch from eded2c4 to d4b9e08 Compare June 18, 2025 09:01

Lingxi Chen added 7 commits June 25, 2025 14:35

Add autotagging rule integration tests

fcf6bb2

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

updated changelog

e7f9e6b

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

revised based on comments

08ef80b

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

removed mocks

37d2ec2

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

fixed setup issue

274cd45

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

deleted unecessary changes

9f4dcda

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

deleted blank line

fceb87a

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

Lindsay-00 force-pushed the rule_IT branch from 555c1ac to fceb87a Compare June 25, 2025 21:35

trying to fix Unknown feature type

17a3568

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

fix unknown feature error

f331044

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

delete print statement

650ea46

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

opensearch-ci-bot mentioned this pull request Jun 30, 2025

[AUTOCUT] Gradle Check Flaky Test Report for PemTrustConfigTests #17745

Open

ruai0511 reviewed Jun 30, 2025

View reviewed changes

Lingxi Chen added 3 commits June 30, 2025 17:37

added verifying whether the query has been tagged with the group ID

681f406

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

revert unnecessay changes

69ef875

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

fixed changelog

f09deac

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

fix precommit error

b46987f

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

triggering tests

5f5ee47

Signed-off-by: Lingxi Chen <lingxich@amazon.com>

This was referenced Sep 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SecureReactorNetty4HttpServerTransportTests #17486

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteClusterStateServiceTests #19325

Closed

jainankitk approved these changes Sep 18, 2025

View reviewed changes

Merge branch 'opensearch-project:main' into rule_IT

fbed93b

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Sep 18, 2025

jainankitk merged commit 4b9aa27 into opensearch-project:main Sep 18, 2025
32 checks passed

This was referenced Sep 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for WarmIndexSegmentReplicationIT #18157

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

Lindsay-00 deleted the rule_IT branch September 19, 2025 18:53

opensearch-ci-bot mentioned this pull request Sep 22, 2025

[AUTOCUT] Gradle Check Flaky Test Report for WlmAutoTaggingIT #19354

Closed

cwperks reviewed Sep 25, 2025

View reviewed changes

andrross added a commit to andrross/OpenSearch that referenced this pull request Sep 25, 2025

Revert "[Rule based auto-tagging] Add autotagging rule integration te…

03aed7a

…sts (opensearch-project#18550)" This reverts commit 4b9aa27.

andrross mentioned this pull request Sep 25, 2025

Revert "Add autotagging rule integration tests" #19423

Closed

1 task

andrross added a commit to andrross/OpenSearch that referenced this pull request Sep 25, 2025

Revert "[Rule based auto-tagging] Add autotagging rule integration te…

58c43f0

…sts (opensearch-project#18550)" This reverts commit 4b9aa27. Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross added a commit to andrross/OpenSearch that referenced this pull request Sep 25, 2025

Revert "[Rule based auto-tagging] Add autotagging rule integration te…

e834f66

…sts (opensearch-project#18550)" This reverts commit 4b9aa27. Signed-off-by: Andrew Ross <andrross@amazon.com>

BrewTestBot mentioned this pull request Oct 15, 2025

opensearch 3.3.0 Homebrew/homebrew-core#249627

Merged

[Rule based auto-tagging] Add autotagging rule integration tests #18550

[Rule based auto-tagging] Add autotagging rule integration tests #18550

Conversation

Lindsay-00 commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

codecov bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ruai0511 commented Jun 19, 2025

Uh oh!

github-actions bot commented Jun 21, 2025

Uh oh!

github-actions bot commented Jun 21, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

ruai0511 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

jainankitk left a comment

Choose a reason for hiding this comment

Uh oh!

jainankitk commented Sep 18, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

jainankitk commented Sep 18, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

Uh oh!

cwperks Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Lindsay-00 commented Jun 18, 2025 •

edited

Loading

codecov bot commented Jun 18, 2025 •

edited

Loading