Merge develop branch into main branch

.github/ISSUE_TEMPLATE/bug_template.md

@@ -0,0 +1,33 @@
+---
+name: 🐛 Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: ''
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Plugins**
+Please list all plugins currently enabled.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Host/Environment (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: 🎆 Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/pull_request_template.md

@@ -0,0 +1,15 @@
+### Description
+[Describe what this change achieves]
+
+### Issues Resolved
+[List any issues this PR will resolve]
+
+### Check List
+- [ ] New functionality includes testing.
+  - [ ] All tests pass
+- [ ] New functionality has been documented.
+  - [ ] New functionality has javadoc added
+- [ ] Commits are signed per the DCO using --signoff 
+
+By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
+For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

.github/workflows/CI-workflow.yml

@@ -0,0 +1,62 @@
+name: Build and Test ML
+on:
+  push:
+    branches:
+      - "*"
+  pull_request:
+    branches:
+      - "*"
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up JDK 1.14
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.14
+
+      - name: Checkout ML
+        uses: actions/checkout@v2
+
+      # Publish OpenSearch to local Maven repo for now
+      - name: Checkout OpenSearch
+        uses: actions/checkout@v2
+        with:
+          repository: 'opensearch-project/OpenSearch'
+          path: OpenSearch
+          ref: '1.0'
+      - name: Build OpenSearch
+        working-directory: ./OpenSearch
+        run: ./gradlew publishToMavenLocal -Dbuild.snapshot=false
+
+      # dependencies: common-utils
+      -   name: Checkout common-utils
+          uses: actions/checkout@v2
+          with:
+            ref: '1.0'
+            repository: 'opensearch-project/common-utils'
+            path: common-utils
+      -   name: Build common-utils
+          working-directory: ./common-utils
+          run: ./gradlew publishToMavenLocal -Dopensearch.version=1.0.0
+
+
+      - name: Build with Gradle
+        run: ./gradlew build -Dopensearch.version=1.0.0
+
+      - name: Publish to Maven Local
+        run: ./gradlew publishToMavenLocal -Dopensearch.version=1.0.0
+
+      - name: Multi Nodes Integration Testing
+        run: ./gradlew integTest  -PnumNodes=3
+
+      - name: Upload Coverage Report
+        uses: codecov/codecov-action@v1
+        with:
+          file: ./build/reports/jacoco/test/jacocoTestReport.xml
+          flags: plugin

.github/workflows/dco.yml

@@ -0,0 +1,18 @@
+name: Developer Certificate of Origin Check
+
+on: [pull_request]
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Get PR Commits
+        id: 'get-pr-commits'
+        uses: tim-actions/get-pr-commits@v1.1.0
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+      - name: DCO Check
+        uses: tim-actions/dco@v1.1.0
+        with:
+          commits: ${{ steps.get-pr-commits.outputs.commits }}

CODE_OF_CONDUCT.md

@@ -1,4 +1,25 @@
-## Code of Conduct
-This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
-For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
-opensource-codeofconduct@amazon.com with any additional questions or comments.
+
+This code of conduct applies to all spaces provided by the OpenSource project including in code, documentation, issue trackers, mailing lists, chat channels, wikis, blogs, social media and any other communication channels used by the project.
+
+
+**Our open source communities endeavor to:**
+
+* Be Inclusive: We are committed to being a community where everyone can join and contribute. This means using inclusive and welcoming language.
+* Be Welcoming: We are committed to maintaining a safe space for everyone to be able to contribute.
+* Be Respectful: We are committed to encouraging differing viewpoints, accepting constructive criticism and work collaboratively towards decisions that help the project grow. Disrespectful and unacceptable behavior will not be tolerated.
+* Be Collaborative: We are committed to supporting what is best for our community and users. When we build anything for the benefit of the project, we should document the work we do and communicate to others on how this affects their work.
+
+
+**Our Responsibility. As contributors, members, or bystanders we each individually have the responsibility to behave professionally and respectfully at all times. Disrespectful and unacceptable behaviors include, but are not limited to:**
+
+* The use of violent threats, abusive, discriminatory, or derogatory language;
+* Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, race, political or religious affiliation;
+* Posting of sexually explicit or violent content;
+* The use of sexualized language and unwelcome sexual attention or advances;
+* Public or private harassment of any kind;
+* Publishing private information, such as physical or electronic address, without permission;
+* Other conduct which could reasonably be considered inappropriate in a professional setting;
+* Advocating for or encouraging any of the above behaviors.
+* Enforcement and Reporting Code of Conduct Issues:
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported. [Contact us](mailto:opensource-codeofconduct@amazon.com). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances.

CONTRIBUTING.md

@@ -1,59 +1,4 @@
-# Contributing Guidelines
+## Contributing to this Project
 
-Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
-documentation, we greatly value feedback and contributions from our community.
-
-Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
-information to effectively respond to your bug report or contribution.
-
-
-## Reporting Bugs/Feature Requests
-
-We welcome you to use the GitHub issue tracker to report bugs or suggest features.
-
-When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
-reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
-
-* A reproducible test case or series of steps
-* The version of our code being used
-* Any modifications you've made relevant to the bug
-* Anything unusual about your environment or deployment
-
-
-## Contributing via Pull Requests
-Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
-
-1. You are working against the latest source on the *main* branch.
-2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
-3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
-
-To send us a pull request, please:
-
-1. Fork the repository.
-2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
-3. Ensure local tests pass.
-4. Commit to your fork using clear commit messages.
-5. Send us a pull request, answering any default questions in the pull request interface.
-6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
-
-GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
-[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
-
-
-## Finding contributions to work on
-Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
-
-
-## Code of Conduct
-This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
-For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
-opensource-codeofconduct@amazon.com with any additional questions or comments.
-
-
-## Security issue notifications
-If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
-
-
-## Licensing
-
-See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+OpenSearch is a community project that is built and maintained by people just like **you**.
+[This document](https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md) explains how you can contribute to this and related projects.

DEVELOPER_GUIDE.md

@@ -0,0 +1,65 @@
+- [Developer Guide](#developer-guide)
+    - [Forking and Cloning](#forking-and-cloning)
+    - [Install Prerequisites](#install-prerequisites)
+        - [JDK 14](#jdk-14)
+    - [Setup](#setup)
+    - [Build](#build)
+        - [Building from the command line](#building-from-the-command-line)
+        - [Building from the IDE](#building-from-the-ide)
+        - [Debugging](#debugging)
+
+## Developer Guide
+
+### Forking and Cloning
+
+Fork this repository on GitHub, and clone locally with `git clone`.
+
+### Install Prerequisites
+
+#### JDK 14
+
+OpenSearch components build using Java 14 at a minimum. This means you must have a JDK 14 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 14 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-14`.
+
+### Setup
+
+1. Clone the repository (see [Forking and Cloning](#forking-and-cloning))
+2. Make sure `JAVA_HOME` is pointing to a Java 14 JDK (see [Install Prerequisites](#install-prerequisites))
+3. Launch Intellij IDEA, Choose Import Project.
+
+### Build
+
+This package uses the [Gradle](https://docs.gradle.org/current/userguide/userguide.html) build system. Gradle comes with excellent documentation that should be your first stop when trying to figure out how to operate or modify the build. we also use the OpenSearch build tools for Gradle. These tools are idiosyncratic and don't always follow the conventions and instructions for building regular Java code using Gradle. Not everything in this package will work the way it's described in the Gradle documentation. If you encounter such a situation, the OpenSearch build tools [source code](https://github.com/opensearch-project/OpenSearch/tree/main/buildSrc/src/main/groovy/org/opensearch/gradle) is your best bet for figuring out what's going on.
+
+#### Building from the command line
+
+1. `./gradlew build` builds and tests
+2. `./gradlew :run` launches a single node cluster with ml-commons plugin installed
+3. `./gradlew :integTest` launches a single node cluster with ml-commons plugin installed and runs all integration tests except security
+4. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
+5. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
+
+When launching a cluster using one of the above commands logs are placed in `/build/cluster/run node0/opensearch-<version>/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.
+
+#### Building from the IDE
+
+Currently, the only IDE we support is IntelliJ IDEA.  It's free, it's open source, it works. The gradle tasks above can also be launched from IntelliJ's Gradle toolbar and the extra parameters can be passed in via the Launch Configurations VM arguments.
+
+#### Debugging
+
+Sometimes it's useful to attach a debugger to either the OpenSearch cluster or the integ tests to see what's going on. When running unit tests you can just hit 'Debug' from the IDE's gutter to debug the tests.  To debug code running in an actual server run:
+
+```
+./gradlew :integTest --debug-jvm # to start a cluster and run integ tests
+OR
+./gradlew :run --debug-jvm # to just start a cluster that can be debugged
+```
+
+The OpenSearch server JVM will launch suspended and wait for a debugger to attach to `localhost:8000` before starting the OpenSearch server.
+
+To debug code running in an integ test (which exercises the server from a separate JVM) run:
+
+```
+./gradlew -Dtest.debug :integTest 
+```
+
+The test runner JVM will start suspended and wait for a debugger to attach to `localhost:5005` before running the tests.

LICENSE

@@ -173,3 +173,30 @@
       defend, and hold each Contributor harmless for any liability
       incurred by, or claims asserted against, such Contributor by reason
       of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MAINTAINERS.md

@@ -0,0 +1,8 @@
+# OpenSearch Machine Learning Maintainers
+
+## Maintainers
+| Maintainer | GitHub ID | Affiliation |
+| --------------- | --------- | ----------- |
+| Hanguang Zhang | [zhanghg08](https://github.com/zhanghg08) | Amazon |
+| Jing Zhang | [jngz-es](https://github.com/jngz-es) | Amazon |
+| Weicong Sun | [weicongs-amazon](https://github.com/weicongs-amazon) | Amazon |

NOTICE.txt

@@ -0,0 +1,12 @@
+OpenSearch
+Copyright 2021 OpenSearch Contributors
+
+This product includes software developed by
+Elasticsearch (http://www.elastic.co).
+Copyright 2009-2018 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).
+
+This product includes software developed by
+Oracle Labs (https://tribuo.org/).