[BUG] Segment-based replication / Remote Store is not compatible with Lucene 9.12 / 10

[reta]

Describe the bug

I have been working on routine Apache Lucene 9.12 update (snapshot) but run into show stopper:

                java.lang.WrongThreadException: Attempted access outside owning thread                                                                                                                                                                                                                                                                
                    at java.base/jdk.internal.foreign.MemorySessionImpl.wrongThread(MemorySessionImpl.java:315)                                                                                                                                                                                                                                       
                    at java.base/jdk.internal.misc.ScopedMemoryAccess$ScopedAccessError.newRuntimeException(ScopedMemoryAccess.java:113)                                                                                                                                                                                                              
                    at java.base/jdk.internal.foreign.MemorySessionImpl.checkValidState(MemorySessionImpl.java:219)                                                                                                                                                                                                                                   
                    at java.base/jdk.internal.foreign.ConfinedSession.justClose(ConfinedSession.java:83)                                                                                                                                                                                                                                              
                    at java.base/jdk.internal.foreign.MemorySessionImpl.close(MemorySessionImpl.java:242)                                                                                                                                                                                                                                             
                    at java.base/jdk.internal.foreign.MemorySessionImpl$1.close(MemorySessionImpl.java:88)                                                                                                                                                                                                                                            
                    at org.apache.lucene.store.MemorySegmentIndexInput.close(MemorySegmentIndexInput.java:514)                                                                                                                                                                                                                                        
                    at org.apache.lucene.tests.store.MockIndexInputWrapper.close(MockIndexInputWrapper.java:81)                                                                                                                                                                                                                                       
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)                                                                                        
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
                    at org.opensearch.indices.replication.SegmentFileTransferHandler$1$1.close(SegmentFileTransferHandler.java:111)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
                    at org.opensearch.indices.replication.SegmentFileTransferHandler$1.close(SegmentFileTransferHandler.java:165)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
                    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:102)
                    at org.opensearch.indices.recovery.MultiChunkTransfer.onCompleted(MultiChunkTransfer.java:170)
                    at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:144)
                    at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98)
                    at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131)
                    at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119)
                    at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97)
                    at org.opensearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:109)
                    at org.opensearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:151)
                    at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
                    at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)
                    at org.opensearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:457)
                    at org.opensearch.indices.recovery.AsyncRecoveryTarget.lambda$writeFileChunk$6(AsyncRecoveryTarget.java:170)
                    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:923)
                    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
                    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
                    at java.base/java.lang.Thread.run(Thread.java:1583)

The issue comes from the fact than Apache Lucene uses FF&MI APIs and more specifically, MemorySegmentIndexInput which is backed by confined memory segment that is not supposed to be shared between multiple threads.

At this moment, I don't not know the exact solution to this, more eyes / minds would be certainly beneficial.

Related component

Storage:Remote

To Reproduce

See please #15333

Expected behavior

The tests should be passing.

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• JDK 21 / 22

Additional context
See please #15333

[reta]

@andrross @mch2 @msfroh tagging you folks, interesting problem that needs to be solved.

[mch2]

@reta Thanks for raising. From the trace looks like this applies to any node-node file copy which would include peer recovery.

[reta]

@reta Thanks for raising. From the trace looks like this applies to any node-node file copy which would include peer recovery.

It might be, the presence of

                    at org.opensearch.indices.replication.SegmentFileTransferHandler$1$1.close(SegmentFileTransferHandler.java:111)

specifically points out to segment replication but to your point, it is not limited to it, thanks @mch2 !

[mch2]

yep thats right the transfer handler is shared between both functions, will take a look at this

[mch2]

Ok this is because we are using IOContext.READONCE to create inputs and then read/send chunks of the files async. This context now enables a requirement that the input is consumed and closed in the same thread within MemorySegmentIndexInput. There is also enforcement within MockDirectoryWrapper that Segments_n files open with a READONCE context, so will need to send that separately or re-create the input if it has more than 1 chunk, and open the rest of the files with READ context.

[mch2]

Thought maybe we could get by with clones on the original inputs bc the clones themselves aren't closed but the creation of them ensures the original is confined via buildSlice here, this prevents us from creating clones from separate threads.

[reta]

Ok this is because we are using IOContext.READONCE to create inputs and then read/send chunks of the files async

I also suspected this but changing to READ was not helping, at least with MockDirectoryWrapper (I will post the error shortly, just far from laptop now).