Automate publishing plugins zips to artifacts repository

[VachaShah]

We should be able to predictably get older releases of opensearch project plugins from an artifact repository. This would be needed to introduce a better way for scalability to test bwc for all releases. This came up in opensearch-project/anomaly-detection#158.

Acceptance Criteria:

Plugin zips should be downloadable from htts://artifacts.opensearch.org/......

[peternied]

[Triage] Plugin releases as zips will be available for only distribution builds, no out of band releases.

[peterzhuamazon]

[Grooming] We need to design this thing so that it installed like the native plugin, contrary to ODFE setup where user need the direct link to the zips.

[abhinavGupta16]

Proposal

There are 2 ways to accomplish this -

1. We can add functionality to install the non-native plugins by only specifying the name (like for native plugins).
2. We provide links and add appropriate documentation on the website such as this to use links and not names to download the plugins.

Approach 1

This approach would require a change in opensearch core since we can only install plugins from predefined list, which is generated on the fly (link) from the plugins listed here

Following are the contents file generated on the fly at the time of build -

(22-01-31 22:23:03) <0> [/tmp/tmpuj_b8b1g/OpenSearch]
% find . -name plugins.txt
./server/build/generated-resources/plugins.txt
./server/build/resources/main/plugins.txt

(22-01-31 22:23:14) <0> [/tmp/tmpuj_b8b1g/OpenSearch]
% cat ./server/build/generated-resources/plugins.txt
analysis-icu
analysis-kuromoji
analysis-nori
analysis-phonetic
analysis-smartcn
analysis-stempel
analysis-ukrainian
discovery-azure-classic
discovery-ec2
discovery-gce
ingest-attachment
mapper-annotated-text
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-hdfs
repository-s3
store-smb
transport-nio%

(22-01-31 22:23:20) <0> [/tmp/tmpuj_b8b1g/OpenSearch]
% cat ./server/build/resources/main/plugins.txt
analysis-icu
analysis-kuromoji
analysis-nori
analysis-phonetic
analysis-smartcn
analysis-stempel
analysis-ukrainian
discovery-azure-classic
discovery-ec2
discovery-gce
ingest-attachment
mapper-annotated-text
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-hdfs
repository-s3
store-smb
transport-nio%

Approach 2

Second approach, we can provide urls for these plugins for download (would require updating existing documentation, example link from the plugins listed here).
Question with approach 2 - What should be the artifact path for the url?

Question for discussion - Which approach do should we follow?
In my opinion, I like the first approach better since it standardizes how we install plugins and eliminates the need to know the download urls.
Thoughts?

[bbarani]

For # 2. The plugins are already available at ci.opensearch.org and we can install it now using the below command and the plugin (zip) artifacts links are present inside the manifest of every distribution.

./opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-alerting-1.2.3.0.zip

I prefer a clean solution for plugin installation (both native and standalone plugin) using same plugin install command for both native and standalone plugins.

[bbarani]

@VachaShah @mch2 @adnapibar @nknize Please provide your inputs on this proposal.

[dblock]

tl;dr For bcw we need to know the released version of the plugin - what's that URL on artifacts.opensearch.org for anomaly-detection.zip and analysis-smartcn.zip? If we're not uploading the former, then I would upload it next to the latter and fix opensearch-project/anomaly-detection#158 to point to those URLs instead of downloading by name.

To enable downloading by name of both plugins, I would modify OpenSearch not to hard-code the plugin names, but to ship with a configurable set of plugin repositories and default to artifacts.opensearch.org. Generating the list of plugins at build time and hard-coding that list seems like unnecessary over-engineering. Installation would iterate through these repositories in order and if the plugin is not found, fail in the same way maven fails saying "i couldn't find this in all these locations", instead of checking a list first.

[abhinavGupta16]

Awaiting response from teams to proceed with this issue further.

[prudhvigodithi]

Hey @VachaShah can this issue be closed also just a thought once this META ISSUE is resolved, we can expect the plugins to be in maven repo, we can fetch them during runtime using maven coordinates for BWC tests during runtime. With this we need not publish again to artifacts.opensearch.org.

[dblock]

Since we publish several ZIPs to artifacts.opensearch.org, we should also publish the other zips there. The maven use-case is very specific for those that use maven as their dependency/build system.

[zelinh]

I'm closing this issue here as we now publish plugin zips to maven. We could also find plugin zips in our ci.opensearch.org with latest index with specific version.
Feel free to reopen this if you believe this is still needed.