[BUG] Flaky integ test NormalizationProcessorIT.testResultProcessor_whenMultipleShardsAndNoMatches_thenSuccessful, NormalizationProcessorIT.testResultProcessor_whenDefaultProcessorConfigAndQueryMatches_thenSuccessful

[martin-gaievski]

What is the bug?

Flaky integ test

How can one reproduce the bug?

In github CI for the plugin you can see that tests for windows are not stable and failing at random.

Example of such failed run:

OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenMultipleShardsAndNoMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenDefaultProcessorConfigAndQueryMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17


Suite: Test class org.opensearch.neuralsearch.processor.NormalizationProcessorIT
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenMultipleShardsAndNoMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenMultipleShardsAndNoMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17
  2> java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertNotNull(Assert.java:713)
        at org.junit.Assert.assertNotNull(Assert.java:723)
        at org.opensearch.neuralsearch.common.BaseNeuralSearchIT.uploadModel(BaseNeuralSearchIT.java:167)
        at org.opensearch.neuralsearch.common.BaseNeuralSearchIT.prepareModel(BaseNeuralSearchIT.java:205)
        at org.opensearch.neuralsearch.processor.NormalizationProcessorIT.setUp(NormalizationProcessorIT.java:62)

    org.opensearch.client.ResponseException: method [DELETE], host [http://localhost:9200], URI [/_search/pipeline/phase-results-pipeline], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"pipeline [phase-results-pipeline] is missing"}],"type":"resource_not_found_exception","reason":"pipeline [phase-results-pipeline] is missing"},"status":404}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:586)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:559)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.deleteSearchPipeline(BaseNeuralSearchIT.java:713)
        at app//org.opensearch.neuralsearch.processor.NormalizationProcessorIT.tearDown(NormalizationProcessorIT.java:69)
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenDefaultProcessorConfigAndQueryMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.processor.NormalizationProcessorIT.testResultProcessor_whenDefaultProcessorConfigAndQueryMatches_thenSuccessful" -Dtests.seed=DABD2B28C8C32DCF -Dtests.security.manager=false -Dtests.locale=pt -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17
  2> java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertNotNull(Assert.java:713)
        at org.junit.Assert.assertNotNull(Assert.java:723)
        at org.opensearch.neuralsearch.common.BaseNeuralSearchIT.uploadModel(BaseNeuralSearchIT.java:167)
        at org.opensearch.neuralsearch.common.BaseNeuralSearchIT.prepareModel(BaseNeuralSearchIT.java:205)
        at org.opensearch.neuralsearch.processor.NormalizationProcessorIT.setUp(NormalizationProcessorIT.java:62)

    org.opensearch.client.ResponseException: method [DELETE], host [http://localhost:9200], URI [/_search/pipeline/phase-results-pipeline], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"pipeline [phase-results-pipeline] is missing"}],"type":"resource_not_found_exception","reason":"pipeline [phase-results-pipeline] is missing"},"status":404}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:586)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:559)
        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.deleteSearchPipeline(BaseNeuralSearchIT.java:713)
        at app//org.opensearch.neuralsearch.processor.NormalizationProcessorIT.tearDown(NormalizationProcessorIT.java:69)
  2> NOTE: leaving temporary files on disk at: /tmp/tmpbvasmums/neural-search/build/testrun/integTest/temp/org.opensearch.neuralsearch.processor.NormalizationProcessorIT_DABD2B28C8C32DCF-001
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=692, maxMBSortInHeap=6.965354608169179, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=pt, timezone=SystemV/PST8PDT
  2> NOTE: Linux 6.1.49-70.116.amzn2023.x86_64 amd64/Eclipse Adoptium 17.0.3 (64-bit)/cpus=16,threads=6,free=425977016,total=536870912
  2> NOTE: All tests run in this JVM: [NeuralSearchIT, ValidateDependentPluginInstallationIT, NeuralQueryEnricherProcessorIT, NormalizationProcessorIT]

36 tests completed, 2 failed

=== Standard output of node `node{::integTest-0}` ===

=== Standard error of node `node{::integTest-0}` ===

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> There were failing tests. See the report at: file:///tmp/tmpbvasmums/neural-search/build/reports/tests/integTest/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 4m 10s

What is the expected behavior?

Test results are stable

What is your host/environment?

Pre-released build of 2.11

[ArnoldTingSu]

tackling this issue today

[navneet1v]

@ArnoldTingSu any update on this issue?

[navneet1v]

@ArnoldTingSu any update on this issue?

[vibrantvarun]

The fix is merged now, therefore closing the issue.