[SPARK-17740] Spark tests should mock / interpose HDFS to ensure that streams are closed

[ericl]

What changes were proposed in this pull request?

As a followup to SPARK-17666, ensure filesystem connections are not leaked at least in unit tests. This is done here by intercepting filesystem calls as suggested by @JoshRosen . At the end of each test, we assert no filesystem streams are left open.

This applies to all tests using SharedSQLContext or SharedSparkContext.

How was this patch tested?

I verified that tests in sql and core are indeed using the filesystem backend, and fixed the detected leaks. I also checked that reverting #15245 causes many actual test failures due to connection leaks.

[rxin]

Thanks - this is super useful!

[rxin]

cc @srowen want to review this?

[SparkQA]

Test build #66143 has finished for PR 15306 at commit 1d80d2e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait SharedSQLContext extends SQLTestUtils with BeforeAndAfterEach

[SparkQA]

Test build #66155 has finished for PR 15306 at commit 401b447.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Merging in master. Thanks!

[rxin]

btw @davies this actually already checks driver side, so we are good here.

[marmbrus]

This is good thing for us to test. However, If you throw an exception in your test that opens a file, it seems that it swallows the exception just telling you that you are leaking files. Is there anyway we could make this less confusing?

[ericl]

Hm, we could move the actual throw to the afterAll(), that would cause a suite abort instead but presumably leave the test errors intact.