Skip to content

[SPARK-12048][SQL] Part 2 Prevent to close JDBC resources twice #10320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-12048][SQL] Part 2 Prevent to close JDBC resources twice #10320

wants to merge 1 commit into from

Conversation

tedyu
Copy link
Contributor

@tedyu tedyu commented Dec 16, 2015

PR #10101 dealt with sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala

This PR applies the same technique on JdbcRDD.scala

@SparkQA
Copy link

SparkQA commented Dec 16, 2015

Test build #47787 has finished for PR 10320 at commit 522afb7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * trait JobSubmitter\n * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n * case class AttachCompletedRebuildUI(appId: String)\n * case class WrapOption(child: Expression, optType: DataType)\n

@tedyu
Copy link
Contributor Author

tedyu commented Dec 16, 2015

@andrewor14 @zsxwing
Please take a look

@@ -100,6 +101,7 @@ class JdbcRDD[T: ClassTag](
}

override def close() {
if (closed) return
try {
if (null != rs) {
rs.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, we could call rs = null after it is closed, also for stmt and conn in the same way to make this function is reenterable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's another approach, though the other implementation opted for a close flag

@zsxwing
Copy link
Member

zsxwing commented Dec 16, 2015

I don't think we need this patch. I saw the following comments in javadoc

     * Calling the method <code>close</code> on a <code>Connection</code>
     * object that is already closed is a no-op.

     * Calling the method <code>close</code> on a <code>Statement</code>
     * object that is already closed has no effect.

     * Calling the method <code>close</code> on a <code>ResultSet</code>
     * object that is already closed is a no-op.

@srowen
Copy link
Member

srowen commented Dec 16, 2015

Javadoc of which? the JDK interfaces? I think the issue was that SQLite didn't actually seem to like being closed twice in practice.

@zsxwing
Copy link
Member

zsxwing commented Dec 16, 2015

Javadoc of which? the JDK interfaces? I think the issue was that SQLite didn't actually seem to like being closed twice in practice.

Forgot to read the JIRA at first. Didn't know that SQLite doesn't follow the Javadoc of the JDK interfaces. Sorry

@tedyu
Copy link
Contributor Author

tedyu commented Dec 16, 2015

I compared JDBCRDD.scala with JdbcRDD.scala
From what I can tell according to the usage of java.sql.Connection and java.sql.ResultSet, the proposed change is needed

@tedyu
Copy link
Contributor Author

tedyu commented Dec 17, 2015

@zsxwing @srowen
Anything I need to do for this PR ?

Thanks

@zsxwing
Copy link
Member

zsxwing commented Dec 17, 2015

Looks NextIterator already has a closed field.

@tedyu
Copy link
Contributor Author

tedyu commented Dec 17, 2015

Looks like the case is covered by NextIterator already

@tedyu tedyu closed this Dec 17, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants