[BUG] Failing unit tests upon integration with 1.0.0

[thalurur]

Describe the bug
IndexManagement plugin unit tests are failing after depending on 1.0.0 version of OpenSearch built using latest commit in main branch commit

Seeing a lot of java.lang.NoClassDefFoundError errors.

Sample errors

java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.common.logging.HeaderWarning
        at org.opensearch.test.OpenSearchTestCase.before(OpenSearchTestCase.java:383)
        at jdk.internal.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)

To Reproduce:

• Pull the OpenSearch main branch locally and do ./gradlew publishToMavenLocal
• Change the version from 7.10.3 to 1.0.0 in IndexManagement
• Run ./gradlew test and see the errors. If I revert to using 7.10.3 no errors are seen

[tlfeng]

Hi @thalurur,

I didn't manage to reproduce the test failure. 🤔

There are many dependencies of Index Management plugin: OpenSearch, Job Scheduler SPI, notification (in Alerting pluign) and common-util, they are all come from my local maven repository, through the guide here: https://github.com/opensearch-project/opensearch-plugins#publish-opensearch-to-maven-local

My steps:

1. Use ./gradlew publishToMavenLocal to generate OpenSearch, Job Scheduler SPI and common-util. (in main branch with the commits until 04/21/2021)
2. Apply the changes of build.gradle from above guide to Alerting plugin to make notification jar able to be published to local repository.
3. Apply the changes of build.gradle from above guide to Index Management plugin to make it retrieve dependencies from local repository.
4. Replace ExceptionsHelper.convertToElastic with ExceptionsHelper.convertToOpenSearchException in both code of Alerting plugin and Index Management plugin.
5. Changing version numbers of the above dependencies in build.gradle correspondingly (in both code of Alerting plugin and Index Management plugin).
6. Run ./gradlew test in directory of Index Management plugin.

Could you try again with all the dependencies in the latest version? Thank you!

I compared the test result of version 1.15.0.0 and 1.13.2.0 of index management plugin, they are almost the same.

$ ./gradlew test

> Configure project :
mixed cluster flag: false
plugin in the node: /Users/ftianli/github/openserch_plugin/index-management/src/test/resources/job-scheduler/opendistro-job-scheduler-1.15.0.0-SNAPSHOT.zip
plugin in the node: /Users/ftianli/github/openserch_plugin/index-management/build/distributions/opensearch-index-management-1.15.0.0-SNAPSHOT.zip
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.7
  OS Info               : Mac OS X 10.16 (x86_64)
  JDK Version           : 14 (AdoptOpenJDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/adoptopenjdk-14.jdk/Contents/Home
  Random Testing Seed   : 957C69D898AC6587
  In FIPS 140 mode      : false
=======================================

> Task :compileKotlin
...

> Task :compileTestKotlin
...

BUILD SUCCESSFUL in 1m 1s
7 actionable tasks: 7 executed

[thalurur]

@tlfeng thanks for the update.

However, its interesting that you aren't seeing errors. I have done the same thing as mentioned.

• Publish OpenSearch 1.0.0 locally
• Publish Alerting built using OpenSearch 1.0.0 locally
• Publish job-scheduler locally
• Publish common-utils locally

Change index-management to use 1.0.0 version of opensearch and to use local maven as source for all dependencies.

And running the test again I see the same errors as yesterday

I have all my local changes in this PR -opensearch-project/index-management#5. Let me also try to run on a different machine (since you were able to make it run)

Also there is a similar issue (NoClassDefFoundError) for AD - opensearch-project/anomaly-detection#15

./gradlew test
Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details

> Configure project :
mixed cluster flag: false
plugin in the node: /Users/rthaluru/Development/opensource/os/open-index-management/src/test/resources/job-scheduler/opendistro-job-scheduler-1.0.0.0-SNAPSHOT.zip
plugin in the node: /Users/rthaluru/Development/opensource/os/open-index-management/build/distributions/opensearch-index-management-1.0.0.0-SNAPSHOT.zip
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.7
  OS Info               : Mac OS X 10.15.7 (x86_64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-14.0.1.jdk/Contents/Home
  Random Testing Seed   : 8D3B3F62F445807D
  In FIPS 140 mode      : false
=======================================


.....

276 tests completed, 65 failed

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///Users/rthaluru/Development/opensource/os/open-index-management/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 11s
7 actionable tasks: 6 executed, 1 up-to-date

[tlfeng]

Thanks for the additional information. 🙂 I will double check if I have ran the test properly.

[thalurur]

hmm.. interesting I tried on a different host and I see no failures

Followed the same procedure

./gradlew test

> Configure project :
mixed cluster flag: false
plugin in the node: /local/home/rthaluru/Development/open-index-management/src/test/resources/job-scheduler/opendistro-job-scheduler-1.0.0.0-SNAPSHOT.zip
plugin in the node: /local/home/rthaluru/Development/open-index-management/build/distributions/opensearch-index-management-1.0.0.0-SNAPSHOT.zip
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.7
  OS Info               : Linux 5.4.95-47.164.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /local/home/rthaluru/jdk-14.0.2
  Random Testing Seed   : C234D5832D757490
  In FIPS 140 mode      : false
=======================================

.....

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 3s
7 actionable tasks: 7 executed

[thalurur]

Tried another time on my mac where I was seeing issues. I purged everything in local maven repository and rebuilt everything, but still see the same errors. But, atleast good to know that it works on other hosts

[tlfeng]

I see. Looks like I can run all the 267 tests locally. (The one failed is made by design to generate the test report)

I haven't have idea of the cause to the test failure yet, but maybe you could try to clean the cache of Gralde using ./graldew clean and keep the dependencies up to date before running the test. 🙂
(Ok, I believe you should have done those properly...)

But as you mentioned that @saratvemulapalli also saw such test failure in AD plugin, so it can still be an issue.

[thalurur]

Cool, yeah I did do ./gradlew clean . It seems to fail everytime I run on my mac, since it was seen in AD tests probably there is some underlying issue, but not certain

[tlfeng]

Hi @thalurur, when you have time, could you run the integration test for Index Management plugin again with the latest Opensearch code? In opensearch-project/anomaly-detection#15, it is said that "NullPointerException that leads to a ClassDefNotFound exception when running tests in anomaly-detection was fixed in OpenSearch PR#604"
I'm also looking forward to see that solved the test failure for Index Management plugin. 😄

[thalurur]

@tlfeng

Verified that with the fix in place no longer seeing the issue to run tests on my mac. Closing the issue :)