[SPARK-3546] InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[sarutak]

No description provided.

[SparkQA]

QA tests have started for PR 2408 at commit bf29d4a.

• This patch merges cleanly.

[srowen]

This would throw an NPE if an error occurred in new RandomAccessFile or getChannel. I suppose you move new RandomAccessFile(file, "r").getChannel before the try block. Before that method returns, there is no FileChannel that successfully opened and therefore needs closing.

[sarutak]

Originally I was going to check channel is null or not, but I forgot at previous PR.
Now I've modified.

[SparkQA]

QA tests have started for PR 2408 at commit b37231a.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2408 at commit bf29d4a.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have finished for PR 2408 at commit b37231a.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ArrayConstructor extends net.razorvine.pickle.objects.ArrayConstructor
  • class SCCallSiteSync(object):

[SparkQA]

QA tests have started for PR 2408 at commit 5f63f67.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2408 at commit 5f63f67.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

doesn't this close the inputstream prematurely? Note that the 3ard argument to results is passed in as a closure so it is lazy.

BTW in my new refactoring of this, there is a place where we should explicitly close the streams:

https://github.com/apache/spark/pull/2330/files#diff-27109eb30a77542d377c936e0d134420R295

[sarutak]

@rxin Exactly, it's not make sense and I noticed the InputStream is closed via NextIterator#close.
So, I revert this part of change.

[rxin]

Jenkins, test this please.

[sarutak]

test this please.

[SparkQA]

QA tests have started for PR 2408 at commit 074781d.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2408 at commit 074781d.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks. Merging in master.