Blog Post: OpenSearch plugin zips now in Maven repo #930
Conversation
Signed-off-by: pgodithi <pgodithi@amazon.com>
Signed-off-by: pgodithi <pgodithi@amazon.com>
Signed-off-by: pgodithi <pgodithi@amazon.com>
elfisher
changed the title
|
Adding @setiah |
|
@setiah @elfisher @krisfreedain Can you please review and provide your comments? |
|
@prudhvigodithi @bbarani - this is great. Have you taken it through the editorial process with @natebower? |
Hey @krisfreedain I believe no, can you please guide me on that :) |
|
@prudhvigodithi Just fill out and submit the OpenSearch Editorial Request form in Asana. If you have never logged in to Asana before, the Log in to Asana page appears. Enter your Amazon email address and click Continue to sync your Amazon email with Asana. Once you are onboarded, fill in the required fields and click Submit to create an editorial request. The request appears in the OpenSearch Editorial Task Tracker project. The editor adds you as a collaborator on the Asana task so that you can view and edit it. You will receive notifications from Asana when you are added to the task as a collaborator, when the editorial review is in progress, and when the editorial review is complete. |
|
@prudhvigodithi @bbarani I'm going to have @Naarcha-AWS review this from a technical writing perspective before I complete my review. Thanks! |
|
Thanks for the update @natebower. |
There was a problem hiding this comment.
Hi @prudhvigodithi.
Per @natebower's request, I went reviewed your blog! It's really excellent to user's have more flexibility in how they install OpenSearch plugins through Maven.
You'll see my suggestions contain a few deletions and additions. I also changed the order each section to:
- Motivation
- Benefits of Maven
- Maven zip publication and gradle
- Consume plugin zips from Maven
My goal was to streamline the information to make it more readable and consumable. That being said, please feel free to modify any of my suggestions to ensure technical accuracy.
| twittercard: | ||
| description: "Details on how to consume OpenSearch plugin zips from maven repo and the process involved in shipping them to the maven repo." | ||
| --- | ||
|
|
There was a problem hiding this comment.
I deleted the Summary section and used that description as the Intro.
| Starting with the release of OpenSearch `2.1.0`, OpenSearch plugin zips are now signed and published to a central Apache Maven [repo](https://repo1.maven.org/maven2/org/opensearch/plugin/). Using the [Release zips Maven Repo](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips Maven Repo](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) URL’s, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster. |
There was a problem hiding this comment.
Sentence Reformation:
Starting with the release of OpenSearch 2.1.0, OpenSearch plugin zips are now signed and published to a central Apache Maven repo. Using the Release zips and Snapshot zips Maven Repo URL’s, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster.
There was a problem hiding this comment.
In "Maven Repo URL's", change to "Maven repo URLs".
| description: "Details on how to consume OpenSearch plugin zips from maven repo and the process involved in shipping them to the maven repo." | ||
| --- | ||
|
|
||
| ### Motivation and Current Challenges: |
There was a problem hiding this comment.
| ### Motivation and Current Challenges: | |
| ## Motivation |
|
|
||
| ### Motivation and Current Challenges: | ||
|
|
||
| The plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, which is not reliable, also the tests executed as part of this build zip cannot be accurate, so eventually for few plugins the local build zip is forced to check in to the plugin repo to facilitate the plugin availability for dependency, this is one of the challenge faced by the community. Coming to the download part of plugins as separate isolated components via a cached mechanism like maven is not possible, with this challenge users had to pre-bake the random built zip to the project. |
There was a problem hiding this comment.
| The plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, which is not reliable, also the tests executed as part of this build zip cannot be accurate, so eventually for few plugins the local build zip is forced to check in to the plugin repo to facilitate the plugin availability for dependency, this is one of the challenge faced by the community. Coming to the download part of plugins as separate isolated components via a cached mechanism like maven is not possible, with this challenge users had to pre-bake the random built zip to the project. | |
| Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime, because plugin zips were not a part of the version-controlled Maven system. The only mechanism for plugin downloads was OpenSearch respective Java jars fetched through Maven coordinates. The Java jars forced users who wanted more control over their OpenSearch plugin configuration to use a random plugin version built on a developer desktop, instead of a more reliable version-controlled plugin. |
There was a problem hiding this comment.
Can we re-form the sentence to match the meaning as follows
Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime, because plugin zips were not a part of the version-controlled Maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, instead of a more reliable version-controlled plugin.
There was a problem hiding this comment.
Remove the commas after "runtime" and "desktop". Capitalize the second instance of "Maven".
|
|
||
| The plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, which is not reliable, also the tests executed as part of this build zip cannot be accurate, so eventually for few plugins the local build zip is forced to check in to the plugin repo to facilitate the plugin availability for dependency, this is one of the challenge faced by the community. Coming to the download part of plugins as separate isolated components via a cached mechanism like maven is not possible, with this challenge users had to pre-bake the random built zip to the project. | ||
|
|
||
| Starting with release `2.1.0` OpenSearch plugin zips are now signed and published to central maven repo, with `groupID` as [org.opensearch.plugin](https://repo1.maven.org/maven2/org/opensearch/plugin/). These zips can now be fetched as individual components by directly downloading using maven coordinates or by clickstream from central maven repo which can be later cached to the desired user local maven repo. The snapshot version of the zips can be fetched from nexus maven repo with the same `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). |
There was a problem hiding this comment.
| Starting with release `2.1.0` OpenSearch plugin zips are now signed and published to central maven repo, with `groupID` as [org.opensearch.plugin](https://repo1.maven.org/maven2/org/opensearch/plugin/). These zips can now be fetched as individual components by directly downloading using maven coordinates or by clickstream from central maven repo which can be later cached to the desired user local maven repo. The snapshot version of the zips can be fetched from nexus maven repo with the same `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). | |
| Furthermore, tests executed against as part ofthe OpenSearch build process from zip were not accurate, as each local build zip had to check against the plugin repo in order to ensure that plugin's availability. These restrictions were challenging to our community, since those wishing to use plugin zips as separate isolated components via a cached mechanism proved to be impossible. |
There was a problem hiding this comment.
This looks really good, small change though
Furthermore, to facilitate the plugin availability as dependency, the plugin zip's are forced to check in to the plugin repo and so tests executed as part of the OpenSearch build process are not accurate. These restrictions were challenging to our community, since those wishing to use plugin zips as separate isolated components via a cached mechanism proved to be impossible.
There was a problem hiding this comment.
Add "a" between "as" and "dependency" ("as a dependency"). Change "zip's" to "zips" (no apostrophe). Change to "...to the plugin repo, so tests...". Remove the comma after "community". Change "since those wishing to use" to "since using".
|
|
||
| Starting with release `2.1.0` OpenSearch plugin zips are now signed and published to central maven repo, with `groupID` as [org.opensearch.plugin](https://repo1.maven.org/maven2/org/opensearch/plugin/). These zips can now be fetched as individual components by directly downloading using maven coordinates or by clickstream from central maven repo which can be later cached to the desired user local maven repo. The snapshot version of the zips can be fetched from nexus maven repo with the same `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). | ||
|
|
||
| ### Maven publish task with custom gradle plugin. |
There was a problem hiding this comment.
I also changed the order of the "Benefits" and "Gradle" sections.
| ### Maven publish task with custom gradle plugin. | |
| ## Benefits of plugin zips in Maven |
| * The development dependent `SNAPSHOT` version zips can be directly fetched from maven using the dependency mechanism. | ||
| * Not required to check in the zips manually to any `src/` folders, as zips can now be fetched using the right `grounID`, `artifactID` and `version`. | ||
| * The published zips to central maven repo are signed with `.asc`, `.md5`, `.sha1`, `.sha256`, `.sha512` extensions. | ||
| * The tests and CI workflows can directly run against the zips fetched from maven repo, instead of requiring a manual download and checking into the project. |
There was a problem hiding this comment.
| * The tests and CI workflows can directly run against the zips fetched from maven repo, instead of requiring a manual download and checking into the project. |
| * The published zips to central maven repo are signed with `.asc`, `.md5`, `.sha1`, `.sha256`, `.sha512` extensions. | ||
| * The tests and CI workflows can directly run against the zips fetched from maven repo, instead of requiring a manual download and checking into the project. | ||
|
|
||
| ### Consume plugin zips from maven: |
There was a problem hiding this comment.
| ### Consume plugin zips from maven: | |
| ## Consume plugin zips from Maven |
| * The tests and CI workflows can directly run against the zips fetched from maven repo, instead of requiring a manual download and checking into the project. | ||
|
|
||
| ### Consume plugin zips from maven: | ||
|
|
There was a problem hiding this comment.
| You can fetch and use OpenSearch plugin zips from Maven in three ways: |
| ``` | ||
|
|
||
| ### Summary: | ||
| Using the maven [Release zips Maven Repo](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips Maven Repo](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) URL’s, the plugin zips can be consumed and used as dependency to build other plugins and can also be fetched as standalone components to install them to the OpenSearch cluster. |
There was a problem hiding this comment.
| Using the maven [Release zips Maven Repo](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips Maven Repo](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) URL’s, the plugin zips can be consumed and used as dependency to build other plugins and can also be fetched as standalone components to install them to the OpenSearch cluster. |
| </dependencies> | ||
| ``` | ||
|
|
||
| ### Summary: |
There was a problem hiding this comment.
| ### Summary: |
_posts/2022-07-24-opensearch-plugin-zips-now-in-maven-repo.markdown
Outdated
Show resolved
Hide resolved
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
|
Hey @Naarcha-AWS, thanks for the review, other than the part which I have updated, rest of your changes LGTM. |
There was a problem hiding this comment.
@Naarcha-AWS @prudhvigodithi @bbarani Please see my comments and let me know if you have any questions. To avoid confusion, it might be best to have @Naarcha-AWS implement my changes. Thanks!
_authors/prudhvigodithi.markdown
Outdated
| photo: '/assets/media/authors/prudhvigodithi.jpg' | ||
| --- | ||
|
|
||
| **Prudhvi Godithi** is a System Enginner at AWS working on the OpenSearch project. |
There was a problem hiding this comment.
Change to "is an AWS System Engineer working on the OpenSearch project."
| twittercard: | ||
| description: "Details on how to consume OpenSearch plugin zips from maven repo and the process involved in shipping them to the maven repo." | ||
| --- | ||
|
|
There was a problem hiding this comment.
In "Maven Repo URL's", change to "Maven repo URLs".
|
|
||
| ### Motivation and Current Challenges: | ||
|
|
||
| The plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, which is not reliable, also the tests executed as part of this build zip cannot be accurate, so eventually for few plugins the local build zip is forced to check in to the plugin repo to facilitate the plugin availability for dependency, this is one of the challenge faced by the community. Coming to the download part of plugins as separate isolated components via a cached mechanism like maven is not possible, with this challenge users had to pre-bake the random built zip to the project. |
There was a problem hiding this comment.
Remove the commas after "runtime" and "desktop". Capitalize the second instance of "Maven".
|
|
||
| The plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates. This forces a user to now to use a dependency plugin zip of some random version built on a developer desktop, which is not reliable, also the tests executed as part of this build zip cannot be accurate, so eventually for few plugins the local build zip is forced to check in to the plugin repo to facilitate the plugin availability for dependency, this is one of the challenge faced by the community. Coming to the download part of plugins as separate isolated components via a cached mechanism like maven is not possible, with this challenge users had to pre-bake the random built zip to the project. | ||
|
|
||
| Starting with release `2.1.0` OpenSearch plugin zips are now signed and published to central maven repo, with `groupID` as [org.opensearch.plugin](https://repo1.maven.org/maven2/org/opensearch/plugin/). These zips can now be fetched as individual components by directly downloading using maven coordinates or by clickstream from central maven repo which can be later cached to the desired user local maven repo. The snapshot version of the zips can be fetched from nexus maven repo with the same `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). |
There was a problem hiding this comment.
Add "a" between "as" and "dependency" ("as a dependency"). Change "zip's" to "zips" (no apostrophe). Change to "...to the plugin repo, so tests...". Remove the comma after "community". Change "since those wishing to use" to "since using".
|
|
||
| ### Maven publish task with custom gradle plugin. | ||
|
|
||
| The publishing of zips to central maven repo had been possible with custom gradle plugin `opensearch.pluginzip`. |
There was a problem hiding this comment.
Add a terminating period to the first list item. In the second list item, should it be "a clickstream"? In the third list item, change "with same Mavin" to "with the same Maven".
|
|
||
| **Using mvn cli:** | ||
|
|
||
| Consume from Central Maven repo: |
There was a problem hiding this comment.
Change to "Consume from the central Maven repo:".
| categories: | ||
| - technical-post | ||
| twittercard: | ||
| description: "Details on how to consume OpenSearch plugin zips from maven repo and the process involved in shipping them to the maven repo." |
There was a problem hiding this comment.
Capitalize both instances of "Maven". Additionally, should it be "Details on how to consume OpenSearch plugin zips from a Maven repo and the process involved in shipping them to a Maven repo."?
| ``` | ||
| mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://repo1.maven.org/maven2 -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0:zip | ||
| ``` | ||
| Consume from Snapshot Maven repo: |
There was a problem hiding this comment.
Add "the" before "Snapshot".
| mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://aws.oss.sonatype.org/content/repositories/snapshots -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0-SNAPSHOT:zip | ||
| ``` | ||
|
|
||
| **Gradle Project: using build.gradle file** |
There was a problem hiding this comment.
Change to "Gradle project: Using the build.gradle file".
| } | ||
| ``` | ||
|
|
||
| **Maven Project: using pom.xml file** |
There was a problem hiding this comment.
Change to "Maven project: Using the pom.xml file".
|
Hey @Naarcha-AWS and @natebower just following up to see if action item pending from my end. :) |
|
@prudhvigodithi @natebower @Naarcha-AWS are we ready to publish? |
|
Hey @natebower and @Naarcha-AWS please let me know if the changes that we discussed looks good? @Naarcha-AWS can you confirm if you implementing those changes ? |
|
@prudhvigodithi @krisfreedain @Naarcha-AWS Once my comments are addressed, you should be good to publish. I thought it might make sense for @Naarcha-AWS to address them, since he did the rewrite, but anyone can feel free to address them in order to expedite the process. |
|
Hey @natebower I can push the suggested changes. |
|
I'm having issues to commit the changes directly proposed by @Naarcha-AWS, just followed up with @Naarcha-AWS, he will raise a PR soon. |
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
|
|
||
| ## Motivation | ||
|
|
||
| Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime because plugin zips were not a part of the version-controlled Maven system. The only mechanism for plugin downloads was each plugins' respective Java jars through Maven coordinates. This system forced users who wanted more control over their OpenSearch plugin configuration to use a dependency plugin zip built on a developer desktop instead of a more reliable version-controlled plugin. |
There was a problem hiding this comment.
Change "plugins'" to "plugin's".
|
|
||
| Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime because plugin zips were not a part of the version-controlled Maven system. The only mechanism for plugin downloads was each plugins' respective Java jars through Maven coordinates. This system forced users who wanted more control over their OpenSearch plugin configuration to use a dependency plugin zip built on a developer desktop instead of a more reliable version-controlled plugin. | ||
|
|
||
| Furthermore, to facilitate the plugin availability as a dependency, tests executed against as part the OpenSearch build process from zip were not accurate, as each local build zip had to find the plugin repo in order to ensure that plugin's availability. These restrictions were challenging to our community since using plugin zips as separate isolated components via a cached mechanism proved to be impossible. |
There was a problem hiding this comment.
Currently reads "...tests executed against as part the...". Please correct. Change "since" to "because".
|
|
||
| With Maven, plugin zips can now be retrieved by: | ||
|
|
||
| - Downloading each plugin directly using their respective Maven coordinates . |
There was a problem hiding this comment.
Remove the space after the period.
|
|
||
| - Downloading each plugin directly using their respective Maven coordinates . | ||
| - Using clickstream from the central Maven repo, which can be cached later to a local Maven repo. | ||
| - Fetching the development `SNAPSHOT` version with same Maven `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). |
There was a problem hiding this comment.
Add "the" before "same".
| Using OpenSearch plugin zips through Maven offers the following benefits: | ||
|
|
||
| - Plugins zip in the central Maven repo are already signed with `.asc`, `.md5`, `.sha1`, `.sha256`, and `.sha512` extensions. | ||
| - Users are no longer required to to check in zips to any `src/` files since zips can be fetched with the right `groupID`, `artifactID`, and `version`. |
There was a problem hiding this comment.
Change "since" to "because".
|
|
||
| - Plugins zip in the central Maven repo are already signed with `.asc`, `.md5`, `.sha1`, `.sha256`, and `.sha512` extensions. | ||
| - Users are no longer required to to check in zips to any `src/` files since zips can be fetched with the right `groupID`, `artifactID`, and `version`. | ||
| - Tests and CI workflows can directly run against zips from the Maven repo instead of requiring a manual download. |
There was a problem hiding this comment.
Define "CI" as "continuous integration (CI)".
|
|
||
| ## Consume plugin in zips | ||
|
|
||
| You can fetch plugin in zips in three different ways: |
There was a problem hiding this comment.
Should "plugin" be "plugins" in both instances, or should there be an article in front of "plugin" in both instances? Change the colon to a period.
|
|
||
| **Using the Maven CLI** | ||
|
|
||
| Consume from central Maven repo: |
There was a problem hiding this comment.
Add "the" before "central".
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
|
Hey @Naarcha-AWS and @natebower, if we are ok, can we go ahead and publish this? |
|
@prudhvigodithi If my comments have been addressed, then we're good to go from my side. |
|
thanks @prudhvigodithi @Naarcha-AWS @natebower - I'll work to push this out tonight/tomorrow morning! |
|
@prudhvigodithi - what are the Gemfile & Gemfile.lock updates for? @AMoo-Miki - can you take a look at these two files? Also, @prudhvigodithi - since this has been in edit for a while - can you update the date to today? (at least inside the file) |
| title: "OpenSearch plugin zips now in Maven repo" | ||
| authors: | ||
| - prudhvigodithi | ||
| date: 2022-07-24 01:01:01 -0700 |
There was a problem hiding this comment.
@prudhvigodithi since this has been in edit for a while - please update this date
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
|
Hey @krisfreedain, I have just updated the date, I can revert those Gemfile changes, but looks like that the update its expecting when configuring the Gem and running the site on my local workspace, let me know I can revert back and look for this later. |
|
Hello @prudhvigodithi -yes, please revert the Gemfile changes. Thanks |
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
Thanks @krisfreedain I have reverted back the Gemfile changes. |
Signed-off-by: pgodithi pgodithi@amazon.com
Description
Blob Post:
OpenSearch plugin zips now in Maven repoIssues Resolved
#929
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.