[BUG] Test org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingExplicitFalse is flaky

[sohami]

Describe the bug

The test org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingExplicitFalse is flaky. It should pass all the time

jun 07, 2024 10:30:45 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
ADVERTENCIA: Uncaught exception in thread: Thread[#12406,Thread-9227,5,TGRP-RemoteStoreMultipartIT]
org.apache.lucene.store.AlreadyClosedException: this Directory is closed
	at __randomizedtesting.SeedInfo.seed([2A9312657A8BBE2A]:0)
	at org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50)
	at org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:175)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$1(RemoteDirectory.java:366)
	at org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getMultipartStreamSupplier$1(RemoteTransferContainer.java:207)
	at org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getTransferPartStreamSupplier$0(RemoteTransferContainer.java:185)
	at org.opensearch.common.StreamContext.provideStream(StreamContext.java:69)
	at org.opensearch.remotestore.multipart.mocks.MockFsAsyncBlobContainer.lambda$asyncBlobUpload$0(MockFsAsyncBlobContainer.java:60)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Stack Trace:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=12406, name=Thread-9227, state=RUNNABLE, group=TGRP-RemoteStoreMultipartIT]
	at __randomizedtesting.SeedInfo.seed([2A9312657A8BBE2A:603559BB7BC19343]:0)
Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed
	at __randomizedtesting.SeedInfo.seed([2A9312657A8BBE2A]:0)
	at app//org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50)
	at app//org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:175)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$1(RemoteDirectory.java:366)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getMultipartStreamSupplier$1(RemoteTransferContainer.java:207)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getTransferPartStreamSupplier$0(RemoteTransferContainer.java:185)
	at app//org.opensearch.common.StreamContext.provideStream(StreamContext.java:69)
	at app//org.opensearch.remotestore.multipart.mocks.MockFsAsyncBlobContainer.lambda$asyncBlobUpload$0(MockFsAsyncBlobContainer.java:60)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)

Related component

Storage:Remote

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingExplicitFalse" -Dtests.seed=2A9312657A8BBE2A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-MX -Dtests.timezone=Africa/Kampala -Druntime.java=21
NOTE: test params are: codec=Asserting(Lucene99): {index_uuid=PostingsFormat(name=Asserting), type=PostingsFormat(name=Asserting)}, docValues:{}, maxPointsInLeafNode=1887, maxMBSortInHeap=7.028352535409868, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-MX, timezone=Africa/Kampala

Expected behavior

Test should pass all the time

Additional Details

CI: https://build.ci.opensearch.org/job/gradle-check/40218/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testAsyncDurabilityWhenRestrictSettingExplicitFalse/

[sachinpkale]

Not able to reproduce on local even after 1000+ iterations on local