Update gradle run to be able to fetch and run external plugins

[mch2]

Description

In trying to test some changes to sql and OpenSearch I found it annoying to have to re-assemble and manually install a plugin every time I made a change to OpenSearch.

This change allows us to install and test plugins that are not shipped with OpenSearch core with gradlew run. You can use it by only specifying the plugin name or adding a specific version to test patches or a released version.

Example :
./gradlew run -PinstalledPlugins['opensearch-job-scheduler'] which will look for opensearch-job-scheduler:<current-opensearch-version>0-SNAPSHOT
or
./gradlew run -PinstalledPlugins['opensearch-job-scheduler:3.2.x.x]

it works fine along with existing plugins:

./gradlew run --debug-server-jvm -PinstalledPlugins="['opensearch-job-scheduler', 'opensearch-sql-plugin', 'repository-s3']"

If you are using a local version of an external plugin like I am the workflow would be:

1. publish core to maven local once so the plugin is built with local changes (if you have api changes).
2. publish the plugin to maven local with changes
3. ./gradlew run as many times as you want, set breakpoints, test things, break things

Related Issues

N/A

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❕ Gradle check result for 77a7d03: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.34%. Comparing base (7abef8c) to head (16271a4).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19428      +/-   ##
============================================
- Coverage     72.93%   72.34%   -0.59%     
+ Complexity    69947    69340     -607     
============================================
  Files          5676     5676              
  Lines        321121   321123       +2     
  Branches      46427    46428       +1     
============================================
- Hits         234195   232310    -1885     
- Misses        68032    69837    +1805     
- Partials      18894    18976      +82     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cwperks]

This is great :). Thank you @mch2!

I wonder if it would be possible in the future to always install plugin dependencies automatically to prevent having to list them out. In either case, this is a big improvement for testing efficiency.

[mch2]

I wonder if it would be possible in the future to always install plugin dependencies automatically to prevent having to list them out. In either case, this is a big improvement for testing efficiency.

yeah we could add a flag or param to fetch a set of bundled plugins. Though I'd imagine keeping that stable with a larger plugin list will be a challenge against snapshot versions, but could provide value in quickly identifying compatibility issues.

[github-actions]

❌ Gradle check result for 16271a4:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

@mch2 Another digression and something that would be separate from this PR, but what do you think about making the opensearch-plugin CLI behave similarly? At least for the official OpenSearch Project plugins, it would be nice to be able to install them by name regardless of whether they are part of the core repository or a separate repository. In other works opensearch-plugin install opensearch-knn should work the same way as opensearch-plugin install repository-s3 works.

[github-actions]

❕ Gradle check result for 16271a4: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mch2]

@mch2 Another digression and something that would be separate from this PR, but what do you think about making the opensearch-plugin CLI behave similarly? At least for the official OpenSearch Project plugins, it would be nice to be able to install them by name regardless of whether they are part of the core repository or a separate repository. In other works opensearch-plugin install opensearch-knn should work the same way as opensearch-plugin install repository-s3 works.

Agreed this would be nice to have, esp given our new snapshot repo doesn't have browse functionality to find the plugins url.