[Searchable Snapshot] IndexInput clones close handling

[aabukhalil]

When Lucene clones IndexInput it will not close it, it will only close the original IndexInput. So we need to keep track of the cloned IndexInputs and close them when they are not used anymore. Scope here is find how to detect the unused IndexInput, is relying on GC and Cleaner is the only option here ? and then implement it

[aabukhalil]

this has been estimated as 5 points

[andrross]

Since the cloned instances are never explicitly closed, then java.lang.ref.Cleaner is the supported mechanism in Java for tracking when those references are no longer in use.

However, given the stated behavior from Lucene:

If you access the cloned IndexInput after closing the original object, any readXXX methods will throw AlreadyClosedException.

Then the question is whether we need to track the cloned instances at all? Per the API contract the cloned instances should become unusable when the original instance is closed, so for the purpose of reference counting and ensuring the underlying cached objects remain alive then it seems possible that we can only track the original instance and ignore the clones (because Lucene ensures that cloned instances are never used beyond the lifetime of the original). We might need to add logic to the clones to fail with the appropriate AlreadyClosedException if they are accessed after the original is closed.

[andrross]

Then the question is whether we need to track the cloned instances at all?

The answer to this question is "yes". With the caching design, and IndexInput instance may end up fetching any number blocks from the cache. Beyond that, the cloned IndexInput instances may fetch different blocks from the cache than the original IndexInput. An alternative approach that doesn't require using the Cleaner could be to have the original IndexInput hold a reference to every block that it or its cloned IndexInputs touched throughout the lifetime of the original, and then release them all when the original is closed. Since we don't control how long that original IndexInput will stay alive, it is better to free the resources using the Cleaner when the clones become unreachable (which may happen while the original IndexInput is still being used, therefore releasing resources sooner than the alternative approach would).