[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

reta · 2024-02-13T23:50:27Z

Describe the bug

The test case org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky:

java.lang.AssertionError: expected:<[1, 1, 0]> but was:<[0, 2, 0]>
	at __randomizedtesting.SeedInfo.seed([A162B5B177457586:2AC595CFBB59CC42]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.opensearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:682)
	at org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation(IndicesRequestCacheIT.java:661)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Search

To Reproduce

/gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.IndicesRequestCacheIT" -Dtests.method="testCacheWithInvalidation {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=A162B5B177457586

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/33724/testReport/junit/org.opensearch.indices/IndicesRequestCacheIT/testCacheWithInvalidation__p0___search_concurrent_segment_search_enabled___true___/

The text was updated successfully, but these errors were encountered:

reta · 2024-02-13T23:51:11Z

@jed326 @sohami another one :(

peternied · 2024-02-14T16:18:17Z

[Triage - attendees 1 2 3 4 5 6]
@reta Thanks for filing

jed326 · 2024-02-14T17:25:33Z

Thanks @reta! At a glance it looks like the failed assertions in this test are related to the request cache which should not have anything specific to concurrent search.

I see there was a similar issue previously #11374 that was closed due to being unable to reproduce the issue. I'll try to take a closer look at this today.

jed326 · 2024-02-16T20:14:22Z

https://build.ci.opensearch.org/job/gradle-check/33829/

This failure occurred with concurrent search disabled, so going to remove this from the concurrent search project.

jed326 · 2024-02-16T20:24:17Z

@sgup432 would you mind checking if this is related to the work being done for #10024?

reta · 2024-02-26T14:21:02Z

Same for org.opensearch.indices.IndicesRequestCacheIT.testCacheWithFilteredAlias {p0={"search.concurrent_segment_search.enabled":"true"}} 👍

java.lang.AssertionError: expected:<[1, 1, 0]> but was:<[0, 2, 0]>
	at __randomizedtesting.SeedInfo.seed([A4730CC52842C528:E3A0E242589B721A]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.opensearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:687)
	at org.opensearch.indices.IndicesRequestCacheIT.testCacheWithFilteredAlias(IndicesRequestCacheIT.java:584)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

These are very frequent offenders and are causing a lot of pain in CI. Related issues: opensearch-project#11374 opensearch-project#12308 opensearch-project#13540 opensearch-project#13503 opensearch-project#13711 opensearch-project#13949 opensearch-project#13437 opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit fbe048f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

reta · 2024-06-19T16:06:58Z

Closing in favour of #14288

…opensearch-project#14082) These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Feb 13, 2024

github-actions bot added the Search Search query, autocomplete ...etc label Feb 13, 2024

github-project-automation bot added this to Search Project Board Feb 13, 2024

github-project-automation bot moved this to 🆕 New in Search Project Board Feb 13, 2024

reta mentioned this issue Feb 13, 2024

Add support of special WrappingSearchAsyncActionPhase so the onPhaseStart() will always be followed by onPhaseEnd() within AbstractSearchAsyncAction #12293

Merged

8 tasks

peternied removed the untriaged label Feb 14, 2024

jed326 added this to Concurrent Search Feb 14, 2024

github-project-automation bot moved this to Todo in Concurrent Search Feb 14, 2024

peternied mentioned this issue Feb 15, 2024

Fix error in RemoteSegmentStoreDirectory when debug logging is enabled #12328

Merged

8 tasks

jed326 removed this from Concurrent Search Feb 16, 2024

andrross added the Search:Performance label Feb 21, 2024

reta mentioned this issue Feb 26, 2024

Bump antlr4 from 4.11.1 to 4.13.1 #12445

Merged

8 tasks

sohami mentioned this issue Mar 16, 2024

[Tiered Caching] Indices Request cache stalekey management #12625

Merged

8 tasks

dblock mentioned this issue Mar 21, 2024

Implement serialization for LinearRing. #12831

Closed

8 tasks

jed326 mentioned this issue Mar 27, 2024

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

Closed

dblock mentioned this issue Apr 30, 2024

Increase cluster publish timeout in flaky test testStaleClusterManage… #13463

Merged

8 tasks

reta mentioned this issue May 14, 2024

[Backport 2.x] Abstracting outbound side of transport (#13293) #13656

Merged

9 tasks

reta mentioned this issue May 27, 2024

[Backport] [2.x] Bump com.google.code.gson:gson in /test/fixtures/hdfs-fixture (#13842) #13847

Merged

andrross mentioned this issue Jun 7, 2024

Mute all flaky IndicesRequestCacheIT tests #14077

Merged

3 tasks

kiranprakash154 assigned kiranprakash154 and sgup432 and unassigned kiranprakash154 Jun 12, 2024

reta closed this as completed Jun 19, 2024

github-project-automation bot moved this from 🆕 New to ✅ Done in Search Project Board Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

reta commented Feb 13, 2024

reta commented Feb 13, 2024

peternied commented Feb 14, 2024

jed326 commented Feb 14, 2024

jed326 commented Feb 16, 2024

jed326 commented Feb 16, 2024

reta commented Feb 26, 2024

reta commented Jun 19, 2024

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

Comments

reta commented Feb 13, 2024

Describe the bug

Related component

To Reproduce

Expected behavior

Additional Details

reta commented Feb 13, 2024

peternied commented Feb 14, 2024

jed326 commented Feb 14, 2024

jed326 commented Feb 16, 2024

jed326 commented Feb 16, 2024

reta commented Feb 26, 2024

reta commented Jun 19, 2024