configure new plugin: ml-commons #1701

ylwu-amzn · 2022-03-03T20:09:37Z

Signed-off-by: Yaliang Wu ylwu@amazon.com

Description

Configure new plugin: ml-commons.

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-03-03T20:13:05Z

Codecov Report

Merging #1701 (cf76dd9) into main (9eb7708) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #1701   +/-   ##
=========================================
  Coverage     94.76%   94.76%           
  Complexity       14       14           
=========================================
  Files           168      168           
  Lines          3512     3512           
  Branches         26       26           
=========================================
  Hits           3328     3328           
  Misses          181      181           
  Partials          3        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9eb7708...cf76dd9. Read the comment docs.

ylwu-amzn · 2022-03-03T21:46:38Z

check (manifests/1.3.0/opensearch-1.3.0.yml) failed

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/tmp4eauaas5/k-NN/build.gradle' line: 51

* What went wrong:
A problem occurred evaluating root project 'opensearch-knn'.
> Failed to apply plugin class 'org.opensearch.gradle.info.GlobalBuildInfoPlugin'.
   > Could not create plugin of type 'GlobalBuildInfoPlugin'.
      > Could not generate a decorated class for type GlobalBuildInfoPlugin.
         > org/gradle/jvm/toolchain/JavaInstallation

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

BUILD FAILED in 11s
2022-03-03 20:29:03 ERROR    CI Manifest check failed
Traceback (most recent call last):
  File "./src/run_ci.py", line 25, in <module>
    sys.exit(main())
  File "./src/run_ci.py", line 21, in main
    CiManifests.from_file(args.manifest, args).check()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_manifest.py", line 21, in check
    self.__check__()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_input_manifest.py", line 36, in __check__
    ci_check_list.check()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_check_list_source.py", line 43, in check
    instance = klass(self.component, self.git_repo, self.target, check.args)
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_check_gradle_properties.py", line 19, in __init__
    self.properties = self.__get_properties()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_check_gradle_properties.py", line 30, in __get_properties
    return PropertiesFile(self.git_repo.output(cmd))
  File "/home/runner/work/opensearch-build/opensearch-build/src/git/git_repository.py", line 78, in output
    return subprocess.check_output(command, cwd=cwd, shell=True).decode().strip()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command './gradlew properties -Dopensearch.version=1.3.0-SNAPSHOT -Dbuild.snapshot=true' returned non-zero exit status 1.
Error: Process completed with exit code 1.

gaiksaya · 2022-03-03T21:53:45Z

Hi @opensearch-project/k-nn , can someone take a look at the above error?
The build for k-nn is failing

gaiksaya · 2022-03-03T22:36:02Z

Hi @ylwu-amzn can you rebase with main? The issue should be fixed now. See #1704
Thanks!

ylwu-amzn · 2022-03-03T22:52:25Z

Hi @ylwu-amzn can you rebase with main? The issue should be fixed now. See #1704 Thanks!

Thanks @gaiksaya , already rebased, waiting for check workflow

ylwu-amzn · 2022-03-03T22:55:06Z

https://github.com/opensearch-project/opensearch-build/runs/5414747375?check_suite_focus=true failed

2022-03-03 22:53:15 INFO     Executing "./gradlew properties -Dopensearch.version=1.3.0-SNAPSHOT -Dbuild.snapshot=true" in /tmp/tmp6e2iv10l/ml-commons
2022-03-03 22:53:31 ERROR    CI Manifest check failed
Traceback (most recent call last):
  File "./src/run_ci.py", line 25, in <module>
    sys.exit(main())
  File "./src/run_ci.py", line 21, in main
    CiManifests.from_file(args.manifest, args).check()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_manifest.py", line 21, in check
    self.__check__()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_input_manifest.py", line 36, in __check__
    ci_check_list.check()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_check_list_source.py", line 44, in check
    instance.check()
  File "/home/runner/work/opensearch-build/opensearch-build/src/ci_workflow/ci_check_gradle_properties_version.py", line 21, in check
    self.properties.check_value("version", self.checked_version)
  File "/home/runner/work/opensearch-build/opensearch-build/src/system/properties_file.py", line 49, in check_value
    raise PropertiesFile.UnexpectedKeyValueError(key, expected, value)
system.properties_file.UnexpectedKeyValueError: Expected to have version='1.3.0.0-SNAPSHOT', but was '1.3.0.0'.

gaiksaya · 2022-03-03T23:00:42Z

https://github.com/opensearch-project/opensearch-build/runs/5414747375?check_suite_focus=true failed

More details on onboarding doc here:
https://github.com/opensearch-project/opensearch-build/blob/main/ONBOARDING.md#onboard-to-build-workflow

Ensure your build.sh reads and passes along both -Dbuild.snapshot= and -Dopensearch.version= flags. Snapshot builds should produce a -SNAPSHOT tagged artifact for example opensearch-plugin-1.1.0.0-SNAPSHOT.zip where a release build of the same component would produce opensearch-plugin-1.1.0.0.zip.

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

ylwu-amzn · 2022-03-04T09:34:22Z

https://github.com/opensearch-project/opensearch-build/blob/main/ONBOARDING.md#onboard-to-build-workflow

Execute ./build.sh to ensure your component builds and all artifacts are correctly placed into ./artifacts/ with correct output names.

./build.sh manifests/1.3.0/opensearch-1.3.0.yml failed to build k-NN. Not sure why, @gaiksaya @jmazanec15 can you help confirm if k-NN can pass ./build.sh now? Is this some known issue?

After removing k-NN from manifests/1.3.0/opensearch-1.3.0.yml , ./build.sh manifests/1.3.0/opensearch-1.3.0.yml ran successfully. Can see opensearch-ml-1.3.0.0.zip in builds/opensearch/plugins

% ll builds/opensearch/plugins
-rw-r--r-- 1 ylwu amazon  11M Mar  4 09:13 opensearch-alerting-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon  13M Mar  4 09:21 opensearch-anomaly-detection-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 503K Mar  4 09:13 opensearch-asynchronous-search-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 5.0M Mar  4 09:22 opensearch-cross-cluster-replication-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 8.1M Mar  4 09:14 opensearch-index-management-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 1.2M Mar  4 09:12 opensearch-job-scheduler-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon  17M Mar  4 09:25 opensearch-ml-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 6.3M Mar  4 09:24 opensearch-observability-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon  59M Mar  4 09:20 opensearch-performance-analyzer-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon 6.8M Mar  4 09:23 opensearch-reports-scheduler-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon  39M Mar  4 09:17 opensearch-security-1.3.0.0.zip
-rw-r--r-- 1 ylwu amazon  18M Mar  4 09:23 opensearch-sql-1.3.0.0.zip

And I see there is maven folder generated under builds/opensearch, for this new ML plugin, we need to publish its client to maven, should we expect similar folder generated like job-scheduler-spi builds/opensearch/maven/org/opensearch/opensearch-job-scheduler-spi/ ? @gaiksaya can you help explain what's this maven folder for and how can we generate ML client maven folder?

ylwu-amzn · 2022-03-04T09:47:04Z

Run ./assemble.sh builds/opensearch/manifest.yml successfully after ./build.sh manifests/1.3.0/opensearch-1.3.0.yml (with k-NN removed from manifests/1.3.0/opensearch-1.3.0.yml).

2022-03-04 09:44:18 INFO     Installed plugins: ['opensearch-alerting', 'opensearch-anomaly-detection', 'opensearch-security', 'opensearch-index-management', 'opensearch-sql', 'opensearch-observability', 'opensearch-cross-cluster-replication', 'opensearch-ml', 'opensearch-performance-analyzer', 'opensearch-job-scheduler', 'opensearch-asynchronous-search', 'opensearch-reports-scheduler']
2022-03-04 09:44:58 INFO     Published /local/home/ylwu/code/os/opensearch-build/dist/opensearch/opensearch-1.3.0-linux-x64.tar.gz.
2022-03-04 09:44:58 INFO     Done.

Install built opensearch/opensearch-1.3.0-linux-x64.tar.gz and start single node cluster on dev desktop, _cat/plugins can show opensearch-ml plugin. Tested train/predict Kmeans, it works well. Checked OpenSearch log, don't see any exception.

gaiksaya · 2022-03-04T16:52:06Z

Hi @dblock , do we need to add build script for ml here: https://github.com/opensearch-project/opensearch-build/tree/main/scripts/components?

tianleh · 2022-03-04T23:48:47Z

Hi @dblock , do we need to add build script for ml here: https://github.com/opensearch-project/opensearch-build/tree/main/scripts/components?

This is the script finder logic https://github.com/opensearch-project/opensearch-build/blob/main/src/paths/script_finder.py#L23

Since it is not added under dir above and also no repo level build script, it will use the default one https://github.com/opensearch-project/opensearch-build/blob/main/scripts/default/opensearch/build.sh

As long as @ylwu-amzn is fine with the default script, there shall be no concern.

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

ylwu-amzn · 2022-03-05T02:39:50Z

Hi @dblock , do we need to add build script for ml here: https://github.com/opensearch-project/opensearch-build/tree/main/scripts/components?

@peterzhuamazon helped explained how maven publishing works and I have published a PR opensearch-project/ml-commons#165 to add build.sh to ml-commons.

Tested locally ./build.sh manifests/1.3.0/opensearch-1.3.0.yml, then can see opensearch-ml-client. @peterzhuamazon can you help confirm if that means all done? Should I change any other place?

% ls maven/org/opensearch/opensearch-ml-client
1.3.0.0  maven-metadata.xml  maven-metadata.xml.md5  maven-metadata.xml.sha1  maven-metadata.xml.sha256  maven-metadata.xml.sha512

And I also ran ./assemble.sh builds/opensearch/manifest.yml, install the built tarball and start local cluster on desktop, it works well, both SQL and MLCommons.

peterzhuamazon

A bit concerned about the naming.
The repo is ml-commons, then there is name opensearch-ml-plugin, and this PR opensearch-project/ml-commons#165 suggest the maven name being opensearch-ml-client.

Can we have a consistent naming convention here?

Thanks.

manifests/1.3.0/opensearch-1.3.0-test.yml

manifests/1.3.0/opensearch-1.3.0.yml

ylwu-amzn requested a review from a team as a code owner March 3, 2022 20:09

gaiksaya mentioned this pull request Mar 3, 2022

Updates 1.3 dashboards manifest to have security-dashboards-plugin reference 1.3 instead of main #1702

Merged

1 task

ylwu-amzn force-pushed the add_ml branch from 96f8dc7 to a9cd50e Compare March 3, 2022 22:39

configure new plugin: ml-commons

4b54389

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

ylwu-amzn force-pushed the add_ml branch from a9cd50e to 4b54389 Compare March 4, 2022 09:05

gaiksaya approved these changes Mar 4, 2022

View reviewed changes

gaiksaya requested a review from dblock March 4, 2022 16:52

move ml-commons before SQL as SQL depends on ml client

cf76dd9

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

tianleh approved these changes Mar 7, 2022

View reviewed changes

peterzhuamazon requested changes Mar 7, 2022

View reviewed changes

manifests/1.3.0/opensearch-1.3.0-test.yml Show resolved Hide resolved

manifests/1.3.0/opensearch-1.3.0.yml Show resolved Hide resolved

peterzhuamazon approved these changes Mar 7, 2022

View reviewed changes

peterzhuamazon merged commit f5cffb9 into opensearch-project:main Mar 7, 2022

ylwu-amzn mentioned this pull request Mar 9, 2022

Publish ML client and common to Maven opensearch-project/ml-commons#122

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

configure new plugin: ml-commons #1701

configure new plugin: ml-commons #1701

ylwu-amzn commented Mar 3, 2022

codecov-commenter commented Mar 3, 2022 •

edited

Loading

ylwu-amzn commented Mar 3, 2022

gaiksaya commented Mar 3, 2022

gaiksaya commented Mar 3, 2022 •

edited

Loading

ylwu-amzn commented Mar 3, 2022

ylwu-amzn commented Mar 3, 2022

gaiksaya commented Mar 3, 2022

ylwu-amzn commented Mar 4, 2022 •

edited

Loading

ylwu-amzn commented Mar 4, 2022 •

edited

Loading

gaiksaya commented Mar 4, 2022

tianleh commented Mar 4, 2022

ylwu-amzn commented Mar 5, 2022

peterzhuamazon left a comment

configure new plugin: ml-commons #1701

configure new plugin: ml-commons #1701

Conversation

ylwu-amzn commented Mar 3, 2022

Description

Check List

codecov-commenter commented Mar 3, 2022 • edited Loading

Codecov Report

ylwu-amzn commented Mar 3, 2022

gaiksaya commented Mar 3, 2022

gaiksaya commented Mar 3, 2022 • edited Loading

ylwu-amzn commented Mar 3, 2022

ylwu-amzn commented Mar 3, 2022

gaiksaya commented Mar 3, 2022

ylwu-amzn commented Mar 4, 2022 • edited Loading

ylwu-amzn commented Mar 4, 2022 • edited Loading

gaiksaya commented Mar 4, 2022

tianleh commented Mar 4, 2022

ylwu-amzn commented Mar 5, 2022

peterzhuamazon left a comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 3, 2022 •

edited

Loading

gaiksaya commented Mar 3, 2022 •

edited

Loading

ylwu-amzn commented Mar 4, 2022 •

edited

Loading

ylwu-amzn commented Mar 4, 2022 •

edited

Loading