FilterDirectory not delegate method copyFrom

[velee]

Description

org.apache.lucene.store.FilterDirectory not delegate method copyFrom, which causes org.apache.lucene.store.HardlinkCopyDirectoryWrapper to be ineffective in the middle layer of the delegate chain.
such as delegate chain:
org.elasticsearch.index.store.Store$StoreDirectory
org.elasticsearch.index.store.ByteSizeCachingDirectory
org.apache.lucene.store.HardlinkCopyDirectoryWrapper
org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory

the wapper org.apache.lucene.store.HardlinkCopyDirectoryWrapper not effect，because org.elasticsearch.index.store.Store$StoreDirectory‘s method copyFrom extends from org.apache.lucene.store.Directory

Version and environment details

No response

[jpountz]

The guidance is that Filter* classes should delegate abstract methods, and not methods that have a default implementation. So I'd rather fix sub classes ofFilterDirectory than FilterDirectory itself.

[velee]

eg. delegate chain: org.apache.lucene.misc.store.ByteWritesTrackingDirectoryWrapper -> org.apache.lucene.misc.store.HardlinkCopyDirectoryWrapper -> org.apache.lucene.store.MMapDirectory

The expectation is to use hardlink instead of copy, but in fact copy is still used, because the call method copyFrom from org.apache.lucene.misc.store.ByteWritesTrackingDirectoryWrapper (it extend from org.apache.lucene.store.Directory).

If the org.apache.lucene.store.FilterDirectory's method copyFrom is decorated, the org.apache.lucene.misc.store.HardlinkCopyDirectoryWrapper's method copyFrom will be used.

[velee]

eg. delegate chain: org.apache.lucene.misc.store.ByteWritesTrackingDirectoryWrapper -> org.apache.lucene.store.TrackingDirectoryWrapper -> org.apache.lucene.store.LockValidatingDirectoryWrapper -> org.apache.lucene.store.MMapDirectory

The org.apache.lucene.store.TrackingDirectoryWrapper and org.apache.lucene.store.LockValidatingDirectoryWrapper's method copyFrom will have no effect.

[jpountz]

Maybe we should make Directory#copyFrom abstract in 11.0 so that FilterDirectory#copyFrom can delegate.

[velee]

Is there any problem delegate with the implementation method? If it is not override in the middle chain, it will finally execute to latest chain of Directory#copyFrom

[jpountz]

Yes, like I wrote above we try to only delegate abstract methods in our Filter* classes. For instance, imagine a FilterDirectory that adds some magic bytes to the beginning of every file. It overrides createOutput to write these bytes and openInput to skip these bytes. The default implementation of copyFrom only relies on createOutput, so it works. If now you start delegating copyFrom in FilterDirectory, it would copy files as-is without adding the magic bytes at the beginning. In general, this causes two problems:

• this may break existing implementations,
• to be correct, all FilterDirectory implementations should review whether they need to override copyFrom, while copyFrom is always correct today.

Your point is valid though, copyFrom being correct is one thing, but if it's inefficient, there's still a bug. In similar situations in the past, we made such methods abstract to convey the expectation that implementations should carefully consider whether/how to implement it.

[velee]

Yes, your thinking is correct, maybe should make Directory#copyFrom abstract, and implementation on BaseDirectory