Refactor checks to ML Indices to return true when MultiTenancy enabled

[brianf-aws]

Description

Currently in environments where MultiTenancy is Enabled and indices are pre-populated outside of the local environment there is a cache invalidation issue. When this occurs there exists a local cold node that checks that the index exists.

clusterService.state().metadata().hasIndex(indexName)

Instead we want to check that multi-tenancy is enabled and return early that we can assert logic that has the above line can continue.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Testing

• ./gradlew spotlessApply
• ./gradlew test

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dbwiddis]

LGTM with a grammar nit and needed test.

[dbwiddis]

Grammar nit: doesFooExists is wrong, either fooExists or doesFooExist should be used. :)

[dbwiddis]

And you need a test added for this method.

[brianf-aws]

Good point, Not sure why the test went through my head. I appreciate your feedback!

[dbwiddis]

Feel free to steal my flow framework test. I got help from Claude anyway.

https://github.com/opensearch-project/flow-framework/blob/76b54dc5836378710b24198ac602410dffd382fb/src/test/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandlerTests.java#L173-L184

[brianf-aws]

Thanks I ended up mocking the responses and check for the actual search when multi-tenancy is disabled

@Test
    public void doesMultiTenantIndexExist_multiTenancyEnabled_returnsTrue() {
        assertTrue(MLIndicesHandler.doesMultiTenantIndexExist(null, true, null));
        MLIndicesHandler mlIndicesHandler = new MLIndicesHandler(clusterService, client, mlFeatureEnabledSetting);
        assertTrue(mlIndicesHandler.doesIndexExists(ML_CONFIG_INDEX));
    }

    @Test
    public void doesMultiTenantIndexExist_multiTenancyDisabledSearchesClusterService_returnsValidSearchResult() {
        assertFalse(MLIndicesHandler.doesMultiTenantIndexExist(clusterService, false, null));

        String sampleIndexName = "test-index";
        when(mlFeatureEnabledSetting.isMultiTenancyEnabled()).thenReturn(false);
        MLIndicesHandler mlIndicesHandler = new MLIndicesHandler(clusterService, client, mlFeatureEnabledSetting);

        when(clusterService.state().metadata().hasIndex(sampleIndexName)).thenReturn(true);
        assertTrue(mlIndicesHandler.doesIndexExists(sampleIndexName));

        when(clusterService.state().metadata().hasIndex(sampleIndexName)).thenReturn(false);
        assertFalse(mlIndicesHandler.doesIndexExists(sampleIndexName));
    }

[dbwiddis]

LGTM with one doc recommendation.

[dbwiddis]

Wasn't aware of this javadoc. I like it... more than "recommended" I'd say it's assumed that when multitenancy is enabled that indices (or tables in DDB) are already created.

[brianf-aws]

Found the error, MLIndices Handler does not have MLFeatureEnabledSetting in time in the MLPlugin class

you need to add that here: https://github.com/opensearch-project/ml-commons/blob/main/plugin/src/main/java/org/opensearch/ml/plugin/MachineLearningPlugin.java#L602

Yep thats what I did and tests pass now :) @dhrubo-os I moved the Setting above

./gradlew test          
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.14.3
  OS Info               : Mac OS X 15.5 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/iflorbri/.sdkman/candidates/java/21.0.5-amzn
  Random Testing Seed   : E58066D372726F36
  Crypto Standard       : any-supported
=======================================

> Task :opensearch-ml-algorithms:compileJava
Note: /Users/iflorbri/IdeaProjects/ml-commons/ml-algorithms/src/main/java/org/opensearch/ml/engine/indices/MLIndicesHandler.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :opensearch-ml-algorithms:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[W811 17:48:00.477036000 LinearAlgebra.cpp:3067] Warning: at::frobenius_norm is deprecated and it is just left for JIT compatibility. It will be removed in a future PyTorch release. Please use `linalg.vector_norm(A, 2., dim, keepdim)` instead (function operator())

> Task :opensearch-ml-plugin:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :opensearch-ml-plugin:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

[Incubating] Problems report is available at: file:///Users/iflorbri/IdeaProjects/ml-commons/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.14.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1m 37s
49 actionable tasks: 7 executed, 42 up-to-date

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.07%. Comparing base (8ef3528) to head (e824173).

Files with missing lines	Patch %	Lines
.../opensearch/ml/action/handler/MLSearchHandler.java	66.66%	0 Missing and 1 partial ⚠️
...ml/action/tasks/CancelBatchJobTransportAction.java	66.66%	0 Missing and 1 partial ⚠️
...search/ml/action/tasks/GetTaskTransportAction.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4089      +/-   ##
============================================
+ Coverage     79.06%   79.07%   +0.01%     
- Complexity     8621     8628       +7     
============================================
  Files           755      755              
  Lines         38352    38361       +9     
  Branches       4287     4287              
============================================
+ Hits          30324    30335      +11     
+ Misses         6220     6218       -2     
  Partials       1808     1808              

Flag	Coverage Δ
ml-commons	79.07% <83.33%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[opensearch-trigger-bot]

The backport to 2.19 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.19 2.19
# Navigate to the new working tree
cd .worktrees/backport-2.19
# Create a new branch
git switch --create backport/backport-4089-to-2.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b8da64257bcf1dcdb953a153ca88c1c1f27a6e1a
# Push it to GitHub
git push --set-upstream origin backport/backport-4089-to-2.19
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.19

Then, create a pull request where the base branch is 2.19 and the compare/head branch is backport/backport-4089-to-2.19.

[dhrubo-os]

@brianf-aws could you please raise a backport PR manually.