[BUG] Test Failure org.opensearch.index.engine.NRTReplicationEngineTests.testUpdateSegments

[xuezhou25]

Describe the bug
Comes from #3568 (comment)

To Reproduce

> Task :server:test

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.engine.NRTReplicationEngineTests.testUpdateSegments" -Dtests.seed=569CC7C1CA942529 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr-RS -Dtests.timezone=Africa/Lome -Druntime.java=17

org.opensearch.index.engine.NRTReplicationEngineTests > testUpdateSegments FAILED
    java.lang.AssertionError: expected:<[_0.cfe, _0.si, _0_1_Asserting_0.dvm, _2.si, _0.cfs, _2.cfe, _2.cfs, _0_1_Asserting_0.dvd, _0_1.fnm, segments_3]> but was:<[_0.si, _0_1_Asserting_0.dvm, _1.cfe, _1.si, _0.cfs, _0_1_Asserting_0.dvd, segments_3, _1.cfs, _1_1_Asserting_0.dvd, _0.cfe, _1_1_Asserting_0.dvm, _0_1.fnm, _1_1.fnm]>
        at __randomizedtesting.SeedInfo.seed([569CC7C1CA942529:36DB290B658B5CC0]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.index.engine.NRTReplicationEngineTests.assertMatchingSegmentsAndCheckpoints(NRTReplicationEngineTests.java:203)
        at org.opensearch.index.engine.NRTReplicationEngineTests.testUpdateSegments(NRTReplicationEngineTests.java:143)

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[reta]

This failure seems to be closely related to [1]. The NRTReplicationReaderManager kind of caches the current set of segment files in constructor but the final set of segment files may change on flush / commit.

apache/lucene#921

[reta]

Could be fixed once #3561 is in

[mch2]

This is happening after the test flushes the primary, the engine.lastCommittedSegmentInfos is not equal to the engine.latestSegmentInfos.

the engine.segments() method computes off of the lastCommittedSegmentInfos. I haven't yet been able to figure out what set of conditions under this seed cause this mismatch.

We can change the test to update the replica with engine.latestSegmentInfos, but then it still fails on comparing engine.segments() between both engines.

[mch2]

Ah ok, I think you are right @reta #3561 could fix this. A merge is kicking in on the primary causing the mismatch.

For the sake of this test we want to assert the replica has been updated with the provided segments, I don't think asserting engine.segments() is necessary if we have already asserted against the passed in segments, will update to fix this.