[BUG] Flaky test org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore

[reta]

Describe the bug

The org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore introduced by #6067 is flaky:

Error Message

java.lang.AssertionError: expected:<1> but was:<0>

Stacktrace

java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([D373B52E9B8323E0:848F02BF30313B1F]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.opensearch.remotestore.RemoteStoreIT.verifyRestoredData(RemoteStoreIT.java:148)
	at org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore(RemoteStoreIT.java:167)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)

To Reproduce
https://build.ci.opensearch.org/job/gradle-check/10883/

Expected behavior
Test should pass all the time

Plugins
Standard

Screenshots
N/A

Host/Environment (please complete the following information):

• CI

Additional context
Add any other context about the problem here.

[reta]

@sachinpkale this is another flaky one :( after #6067

[sachinpkale]

:( Checking, I don't think it is due to #6067 but will check and fix.

[reta]

Thanks @sachinpkale , as far as I can tell, the test [1] was introduced in #6067:

[1] https://github.com/opensearch-project/OpenSearch/pull/6067/files#diff-a0573be9c40dba62016508046e973a49ca40c7d84b26957b82462617bc79a5cdR152

[sachinpkale]

Yes, that is correct. I got confused with the PRs. I thought you were referring to latest changes to this test as per #6230.

@reta I have fixed the test in #6252. Please take a look.

[sachinpkale]

Fixed in #6252

[dreamer-89]

@sachinpkale : Can you please backport the fix to 2.x branch as well ? Found one occurrence #6288

[sachinpkale]

This should be already backported. If it is still failing, I will mute the test.

[sachinpkale]

Not able to reproduce it with the same seed. Also, this is a different issue than the one mentioned in this ticket as the error is different. I am closing this ticket and opening new one. Muting the test.