+opensearch #4818

scottjg · 2024-01-12T00:29:03Z

my attempt at adding the opensearch package. there is an existing homebrew package that i used as a reference. this is my first time contributing to the pantry so please let me know if any changes are needed.

i would also like to contribute building the k-NN plugin and native binaries for opensearch. this is normally shipped with opensearch binaries by the upstream project but annoyingly they don't include the native libraries for macos despite them working fine on the platform. i was hoping someone could maybe point me at an example of including a second distribution tarball.

jhheider

looks good; just some style notes.

jhheider · 2024-01-12T02:53:00Z

projects/opensearch.org/package.yml

+    - gradle -Dbuild.snapshot=false ":distribution:archives:no-jdk-{{hw.platform}}-tar:assemble"
+    - mkdir -p "{{prefix}}"
+    - tar --strip-components=1 -xf distribution/archives/no-jdk-{{hw.platform}}-tar/build/distributions/opensearch-*.tar.gz -C "{{prefix}}"


what a weird way to have to do it; sadly, python poetry is similar.

projects/opensearch.org/package.yml

jhheider · 2024-01-12T04:21:48Z

i would also like to contribute building the k-NN plugin and native binaries for opensearch. this is normally shipped with opensearch binaries by the upstream project but annoyingly they don't include the native libraries for macos despite them working fine on the platform. i was hoping someone could maybe point me at an example of including a second distribution tarball.

you should be able to add it to the install script, assuming you can build all the required parameters. would you like me to wait until you add that to merge?

scottjg · 2024-01-12T04:24:52Z

sure - let me dig into it.

scottjg · 2024-01-12T22:22:12Z

projects/opensearch.org/package.yml

+        ./gradlew publishPluginZipPublicationToZipStagingRepository -Dopensearch.version={{version}} -Dbuild.snapshot=false -Dbuild.version_qualifier=
+        ./gradlew publishPluginZipPublicationToMavenLocal -Dbuild.snapshot=false -Dbuild.version_qualifier= -Dopensearch.version={{version}}
+        mkdir -p ./build/distributions/lib
+        cp -v ./jni/release/libopensearchknn* ./build/distributions/lib


this build generates native libraries with the .jnilib extension. when i was packaging this for my team (pre-pkgx) i had to codesign and notarize the libraries. i know you guys have some infra around this stuff already but i just wanted to make sure since it wasn't a dylib that it would be ok?

it seems to work locally fwiw.

generally, yes, our codesigning logic has worked for over a year. the only package i can see that's needed anything different is: https://github.com/pkgxdev/pantry/blob/main/projects/github.com/sindresorhus/macos-term-size/package.yml

scottjg · 2024-01-13T06:45:06Z

oops, sorry.. didn't see that force push. might've clobbered each other there.

jhheider · 2024-01-13T06:50:38Z

no worries; i just did update-with-rebase

includes knn plugin

scottjg · 2024-01-13T07:04:44Z

great progress here, but still a few issues.

mac x86-64 build: seems to be having some kind of networking issue (maybe?) fetching a manifest
mac m1 build: rpaths are not properly cleaned up in the jnilibs and they fail to load
linux builds: likely similar rpath issues but tests don't run on them

jhheider · 2024-01-13T07:18:18Z

from boost.org:

    # boost.org has libs that end up with name @rpath/libboost_atomic.dylib (offset 24)
    # so we need to add @loader_path to the rpath
    - run: |
        for LIB in *.dylib; do
          install_name_tool -add_rpath @loader_path $LIB
        done
      working-directory: ${{prefix}}/lib
      if: darwin

i'll re-run darwin-x86_64 once the rest finish (though it should have the same issue).

linux doesn't use rpath, so that's not likely to affect them. it uses LD_LIBRARY_PATH, which we set. looks like linux-aarch64 passed (unless you mean it has no tests).

jhheider · 2024-01-13T07:35:22Z

that .meta url is fine, so something is up with JsonSlurper on darwin-x86-64. i wonder if it's relying on a system tool that isn't present. probably have to try one of these to see why:

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.

scottjg · 2024-01-13T07:45:11Z

linux doesn't use rpath, so that's not likely to affect them. it uses LD_LIBRARY_PATH, which we set. looks like linux-aarch64 passed (unless you mean it has no tests).

the real tests i wrote in the yml only run on darwin. opensearch can't run as root (mysql package has a similar comment in it), so i just output the version instead which can at least sanity check the package. the problem is that the knn plugin is lazily loaded so that's not really tested at all on linux.

maybe there's some simple-ish solution munging env with sudo to make that work.

that .meta url is fine, so something is up with JsonSlurper on darwin-x86-64. i wonder if it's relying on a system tool that isn't present. probably have to try one of these to see why:

makes sense. unfortunately past my bedtime so will have to try in the morning.

getting close though! m1 build is totally working end to end.

trying to see what's wrong with jsonslurper

it's too noisy to see what's going on

scottjg · 2024-01-13T19:05:02Z

mysql package has a similar comment in it

err, i meant postgres

pantry/projects/postgresql.org/package.yml

Lines 80 to 85 in fde48d5

    
           test: 
        
             script: | 
        
               # While we'd love a good test like this, `initdb` doesn't run as root, and managing 
        
               # all the environment passthrough with `sudo` is a nightmare. 
        
               if test "{{ hw.platform }}" = "linux"; then

scottjg · 2024-01-13T19:28:41Z

not sure what happened with that JsonSlurper issue but it's gone this morning. i also sanity checked the linux build locally and it seemed to work. i think this is ready for a final review.

jhheider · 2024-01-14T03:33:42Z

looks good. i tried to clean up the various cds with working-directories: to make the flow more clear. however, if i didn't follow it, i might have messed it up. we'll know after it builds.

scottjg · 2024-01-14T04:18:25Z

nice!

jhheider · 2024-01-14T04:21:09Z

And there we go. That was a big one. Thanks @scottjg !

jhheider · 2024-01-14T05:05:51Z

hm, failing to find libc on macos12: https://github.com/pkgxdev/pantry/actions/runs/7516912520/job/20462627197

possibly we need to add an rpath somewhere?

scottjg · 2024-01-14T06:35:04Z

projects/opensearch.org/package.yml

+        zip -r opensearch-knn-{{version}}.0.zip lib/
+        {{prefix}}/bin/opensearch-plugin install --batch file:`pwd`/opensearch-knn-{{version}}.0.zip
+      working-directory: k-NN
+    - run: echo 'export OPENSEARCH_JAVA_OPTS="-Djava.library.path=$OPENSEARCH_HOME/plugins/opensearch-knn/lib $OPENSEARCH_JAVA_OPTS"' >> opensearch-env


if it's happening when opensearch boots, then it shouldn't be loading the knn plugin yet. it might be related to this.

i tried to modify the java library path here so that the native plugin dependencies would be loaded properly. maybe it needs some more paths? hmm...

does it maybe need a runtime.env.OPENSEARCH_JAVA_OPTS: key?

scottjg · 2024-01-14T06:41:03Z

so, i guess it's in opensearch itself where it's trying to call setrlimit which presumably is why it's trying to load libc

java.lang.UnsatisfiedLinkError: 'int org.opensearch.bootstrap.JNACLibrary.setrlimit(int, org.opensearch.bootstrap.JNACLibrary$Rlimit)'
	at org.opensearch.bootstrap.JNACLibrary.setrlimit(Native Method) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.SystemCallFilter.bsdImpl(SystemCallFilter.java:609) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:669) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:281) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:128) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:129) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.11.1.jar:2.11.1]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.11.1.jar:2.11.1]

scottjg · 2024-01-14T07:11:01Z

#4841

jhheider approved these changes Jan 12, 2024

View reviewed changes

scottjg mentioned this pull request Jan 12, 2024

version request: openjdk.org@^17 #4829

Closed

scottjg commented Jan 12, 2024

View reviewed changes

jhheider force-pushed the opensearch branch from c14a729 to 58a4317 Compare January 13, 2024 06:42

+opensearch

f2aa283

includes knn plugin

scottjg force-pushed the opensearch branch from ce8ca07 to f2aa283 Compare January 13, 2024 06:54

opensearch: fix the rpath on the jnilibs

4122e18

scottjg added 3 commits January 13, 2024 10:24

opensearch: add debug lines to the gradle invocation

01acd77

trying to see what's wrong with jsonslurper

opensearch: remove the debug log

bc6cf4b

it's too noisy to see what's going on

opensearch: does it help if we always use clang on mac?

e2a07e0

scottjg added 2 commits January 13, 2024 11:13

opensearch: reshuffle some workarounds on mac

f329229

opensearch: cleanup deps

0e0ebb8

clean up cd > working-directory

e3fe752

jhheider added 3 commits January 13, 2024 22:35

yaml quoting

3a5aee3

more quoting

f7b0d83

fix my fixes

e120945

jhheider merged commit 488f027 into pkgxdev:main Jan 14, 2024

scottjg commented Jan 14, 2024

View reviewed changes

+opensearch #4818

+opensearch #4818

Uh oh!

Conversation

scottjg commented Jan 12, 2024

Uh oh!

jhheider left a comment

Choose a reason for hiding this comment

Uh oh!

jhheider Jan 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jhheider commented Jan 12, 2024

Uh oh!

scottjg commented Jan 12, 2024

Uh oh!

scottjg Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhheider Jan 12, 2024

Choose a reason for hiding this comment

Uh oh!

scottjg commented Jan 13, 2024

Uh oh!

jhheider commented Jan 13, 2024

Uh oh!

scottjg commented Jan 13, 2024

Uh oh!

jhheider commented Jan 13, 2024

Uh oh!

jhheider commented Jan 13, 2024

Uh oh!

scottjg commented Jan 13, 2024

Uh oh!

scottjg commented Jan 13, 2024

Uh oh!

scottjg commented Jan 13, 2024

Uh oh!

jhheider commented Jan 14, 2024

Uh oh!

scottjg commented Jan 14, 2024

Uh oh!

jhheider commented Jan 14, 2024

Uh oh!

jhheider commented Jan 14, 2024

Uh oh!

scottjg Jan 14, 2024

Choose a reason for hiding this comment

Uh oh!

jhheider Jan 14, 2024

Choose a reason for hiding this comment

Uh oh!

scottjg commented Jan 14, 2024

Uh oh!

scottjg commented Jan 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

scottjg Jan 12, 2024 •

edited

Loading