Skip to content

[SPARK-6294] fix hang when call take() in JVM on PythonRDD #4987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-6294] fix hang when call take() in JVM on PythonRDD #4987

wants to merge 1 commit into from

Conversation

davies
Copy link
Contributor

@davies davies commented Mar 12, 2015

The Thread.interrupt() can not terminate the thread in some cases, so we should not wait for the writerThread of PythonRDD.

This PR also ignore some exception during clean up.

cc @JoshRosen @mengxr

@SparkQA
Copy link

SparkQA commented Mar 12, 2015

Test build #28487 has started for PR 4987 at commit 4488f1a.

  • This patch merges cleanly.

@mengxr
Copy link
Contributor

mengxr commented Mar 12, 2015

LGTM pending Jenkins.

@SparkQA
Copy link

SparkQA commented Mar 12, 2015

Test build #28487 has finished for PR 4987 at commit 4488f1a.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28487/
Test FAILed.

@mengxr
Copy link
Contributor

mengxr commented Mar 12, 2015

test this please

@SparkQA
Copy link

SparkQA commented Mar 12, 2015

Test build #28499 has started for PR 4987 at commit 4488f1a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Mar 12, 2015

Test build #28499 has finished for PR 4987 at commit 4488f1a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28499/
Test PASSed.

asfgit pushed a commit that referenced this pull request Mar 12, 2015
@mengxr
[SPARK-6294] fix hang when call take() in JVM on PythonRDD
850e694 
The Thread.interrupt() can not terminate the thread in some cases, so we should not wait for the writerThread of PythonRDD.

This PR also ignore some exception during clean up.

cc JoshRosen mengxr

Author: Davies Liu <davies@databricks.com>

Closes #4987 from davies/fix_take and squashes the following commits:

4488f1a [Davies Liu] fix hang when call take() in JVM on PythonRDD

(cherry picked from commit 712679a)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@asfgit asfgit closed this in 712679a Mar 12, 2015
@mengxr
Copy link
Contributor

mengxr commented Mar 12, 2015

Merged into master and branch-1.3. @davies This cannot be merged cleanly into branch-1.2. Could you send another PR for branch-1.2 and maybe branch-1.1? Thanks!

justinuang pushed a commit to justinuang/spark that referenced this pull request Apr 16, 2015
[SPARK-6294] fix hang when call take() in JVM on PythonRDD
b06616d 
The Thread.interrupt() can not terminate the thread in some cases, so we should not wait for the writerThread of PythonRDD.

This PR also ignore some exception during clean up.

cc JoshRosen mengxr

Author: Davies Liu <davies@databricks.com>

Closes apache#4987 from davies/fix_take and squashes the following commits:

4488f1a [Davies Liu] fix hang when call take() in JVM on PythonRDD
@justinuang justinuang mentioned this pull request Apr 16, 2015
Closed
@justinuang justinuang mentioned this pull request Apr 17, 2015
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@davies @SparkQA @mengxr @AmplabJenkins