Skip to content

[SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 #30447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 #30447

wants to merge 3 commits into from

Conversation

sunchao
Copy link
Member

@sunchao sunchao commented Nov 20, 2020

What changes were proposed in this pull request?

Revert the change in #29959 and don't remove SerializableFileStatus and SerializableBlockLocation.

Why are the changes needed?

In Hadoop 2.7 FileStatus and BlockLocation are not serializable, so we still need the two wrapper classes.

Does this PR introduce any user-facing change?

No

How was this patch tested?

N/A

@sunchao sunchao mentioned this pull request Nov 20, 2020
Closed
@github-actions github-actions bot added the CORE label Nov 20, 2020
@dongjoon-hyun
Copy link
Member

Thank you, @sunchao !

cc @holdenk

@sunchao sunchao changed the title [SPARK-32381][CORE][FOLLOWUP][HADOOP-2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 [SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 Nov 20, 2020
@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36038/

holdenk
holdenk reviewed Nov 20, 2020
View reviewed changes
holdenk
holdenk approved these changes Nov 20, 2020
View reviewed changes
Copy link
Contributor

@holdenk holdenk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending Jenkins, one minor suggestion re: adding a comment.

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36038/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36040/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36040/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Test build #131432 has finished for PR 30447 at commit c4a188c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sunchao
Copy link
Member Author

sunchao commented Nov 20, 2020

Cool the jenkins CI passed and the failure in github CI seems not related. I updated the PR with some comments.

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Test build #131434 has finished for PR 30447 at commit 96d0578.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36048/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36048/

@holdenk
Copy link
Contributor

holdenk commented Nov 20, 2020

Jenkins retest this please.

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Test build #131442 has finished for PR 30447 at commit d4e542e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 21, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36059/

@SparkQA
Copy link

SparkQA commented Nov 21, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36059/

@SparkQA
Copy link

SparkQA commented Nov 21, 2020

Test build #131453 has finished for PR 30447 at commit d4e542e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Nov 21, 2020
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thanks, @sunchao and @holdenk .
This passed at #30447 (comment)
Merged to master to recover Hadoop 2.7 profile.

@sunchao sunchao deleted the SPARK-32381-followup branch November 21, 2020 02:51
@sunchao
Copy link
Member Author

sunchao commented Nov 21, 2020

Thanks @dongjoon-hyun and @holdenk !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@holdenk holdenk holdenk approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees
No one assigned
Labels
CORE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sunchao @dongjoon-hyun @SparkQA @holdenk