Use testclusters when testing with security

[cwperks]

Description

This PR changes the Test with Security workflow to use testclusters instead of the docker image. The integTest task in OpenSearch is meant to be run against an opensearch testcluster (example from ISM) and the current workflow ends up spinning up 2 clusters (one testcluster and 1 cluster w/ docker) unnecessarily.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.61%. Comparing base (41db8c0) to head (101f702).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1414      +/-   ##
============================================
- Coverage     81.62%   81.61%   -0.01%     
+ Complexity     5870     5864       -6     
============================================
  Files           533      533              
  Lines         23669    23669              
  Branches       2374     2374              
============================================
- Hits          19319    19317       -2     
- Misses         3190     3193       +3     
+ Partials       1160     1159       -1     

Flag	Coverage Δ
plugin	81.61% <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amitgalitz]

Thanks for adding this, it can be done in a follow up PR as well but could you also add a link in our developer guide (https://github.com/opensearch-project/anomaly-detection/blob/main/DEVELOPER_GUIDE.md) that we a can now run integTest with -Dsecurity.enabled=true I assume we can also now run ./gradlew run -Dsecurity.enabled=true to run the cluster with security enabled locally?

[cwperks]

Added a new entry to the developer guide on how to run integ tests with security.

I assume we can also now run ./gradlew run -Dsecurity.enabled=true to run the cluster with security enabled locally?

Not with this change. This is only for setting up a testcluster for the integTest task. It should be possible to get this to work, but not with the changes in this PR.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/anomaly-detection/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/anomaly-detection/backport-2.x
# Create a new branch
git switch --create backport/backport-1414-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 03637f1534cea3fa2a6f665fb35cf7b7e52c0e32
# Push it to GitHub
git push --set-upstream origin backport/backport-1414-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/anomaly-detection/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1414-to-2.x.