Skip to content

[SPARK-12049] [CORE] User JVM shutdown hook can cause deadlock at shutdown #10042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-12049] [CORE] User JVM shutdown hook can cause deadlock at shutdown #10042

wants to merge 2 commits into from

Conversation

srowen
Copy link
Member

@srowen srowen commented Nov 30, 2015

Avoid potential deadlock with a user app's shutdown hook thread by more narrowly synchronizing access to 'hooks'

@SparkQA
Copy link

SparkQA commented Nov 30, 2015

Test build #46887 has finished for PR 10042 at commit d22cffa.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class Count(children: Seq[Expression]) extends DeclarativeAggregate\n

vanzin
vanzin reviewed Nov 30, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
while (!hooks.isEmpty()) {
Try(Utils.logUncaughtExceptions(hooks.poll().run()))
var nextHook: SparkShutdownHook = null
while ({nextHook = hooks synchronized { hooks.poll() }; nextHook != null}) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we generally use hooks.synchronized.

@SparkQA
Copy link

SparkQA commented Nov 30, 2015

Test build #46895 has finished for PR 10042 at commit 934f237.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Nov 30, 2015

LGTM.

@SparkQA
Copy link

SparkQA commented Nov 30, 2015

Test build #46899 has finished for PR 10042 at commit 3da8644.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class Count(children: Seq[Expression]) extends DeclarativeAggregate\n

@vanzin
Copy link
Contributor

vanzin commented Dec 1, 2015

Merging (master, 1.6, 1.5).

asfgit pushed a commit that referenced this pull request Dec 1, 2015
@srowen
[SPARK-12049][CORE] User JVM shutdown hook can cause deadlock at shut…
2fc3fce 
…down

Avoid potential deadlock with a user app's shutdown hook thread by more narrowly synchronizing access to 'hooks'

Author: Sean Owen <sowen@cloudera.com>

Closes #10042 from srowen/SPARK-12049.

(cherry picked from commit 96bf468)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
@asfgit asfgit closed this in 96bf468 Dec 1, 2015
asfgit pushed a commit that referenced this pull request Dec 1, 2015
@srowen
[SPARK-12049][CORE] User JVM shutdown hook can cause deadlock at shut…
d78f1bc 
…down

Avoid potential deadlock with a user app's shutdown hook thread by more narrowly synchronizing access to 'hooks'

Author: Sean Owen <sowen@cloudera.com>

Closes #10042 from srowen/SPARK-12049.

(cherry picked from commit 96bf468)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
@srowen srowen deleted the SPARK-12049 branch December 1, 2015 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@srowen @SparkQA @vanzin