[SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM #32487

viirya · 2021-05-09T21:43:56Z

What changes were proposed in this pull request?

This patch proposes to increase the maximum heap memory setting for release build.

Why are the changes needed?

When I was cutting RCs for 2.4.8, I frequently encountered OOM during building using mvn. It happens many times until I increased the heap memory setting.

I am not sure if other release managers encounter the same issue. So I propose to increase the heap memory setting and see if it looks good for others.

Does this PR introduce any user-facing change?

No, dev only.

How was this patch tested?

Manually used it during cutting RCs of 2.4.8.

viirya · 2021-05-09T21:50:30Z

cc @HyukjinKwon @dongjoon-hyun

SparkQA · 2021-05-09T22:53:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42832/

SparkQA · 2021-05-09T22:53:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42832/

dongjoon-hyun · 2021-05-09T23:06:31Z

dev/create-release/release-build.sh

    PYSPARK_VERSION=`echo "$SPARK_VERSION" |  sed -e "s/-/./" -e "s/SNAPSHOT/dev0/" -e "s/preview/dev/"`
    echo "__version__='$PYSPARK_VERSION'" > python/pyspark/version.py

+    export MAVEN_OPTS="-Xmx12000m"


Can we have this globally outside of if statement? Then, it looks like we need only one line addition.

nit: we can say -Xmx12g?

dongjoon-hyun · 2021-05-09T23:06:44Z

Also, cc @srowen

SparkQA · 2021-05-10T00:34:06Z

Test build #138310 has finished for PR 32487 at commit 2d27589.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2021-05-10T04:24:52Z

Getting pretty big! but OK if needed.

SparkQA · 2021-05-10T05:05:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42836/

SparkQA · 2021-05-10T05:05:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42836/

SparkQA · 2021-05-10T06:15:25Z

Test build #138314 has finished for PR 32487 at commit 4098407.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-05-10T06:20:30Z

retest this please

SparkQA · 2021-05-10T07:01:04Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42844/

…d to avoid OOM ### What changes were proposed in this pull request? This patch proposes to increase the maximum heap memory setting for release build. ### Why are the changes needed? When I was cutting RCs for 2.4.8, I frequently encountered OOM during building using mvn. It happens many times until I increased the heap memory setting. I am not sure if other release managers encounter the same issue. So I propose to increase the heap memory setting and see if it looks good for others. ### Does this PR introduce _any_ user-facing change? No, dev only. ### How was this patch tested? Manually used it during cutting RCs of 2.4.8. Closes #32487 from viirya/release-mvn-oom. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 20d3224) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…d to avoid OOM ### What changes were proposed in this pull request? This patch proposes to increase the maximum heap memory setting for release build. ### Why are the changes needed? When I was cutting RCs for 2.4.8, I frequently encountered OOM during building using mvn. It happens many times until I increased the heap memory setting. I am not sure if other release managers encounter the same issue. So I propose to increase the heap memory setting and see if it looks good for others. ### Does this PR introduce _any_ user-facing change? No, dev only. ### How was this patch tested? Manually used it during cutting RCs of 2.4.8. Closes #32487 from viirya/release-mvn-oom. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 20d3224) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9623ccb) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2021-05-10T07:30:33Z

Merged to master/3.1/3.0.

SparkQA · 2021-05-10T09:54:20Z

Test build #138322 has finished for PR 32487 at commit 4098407.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-10T10:00:05Z

Test build #138323 has finished for PR 32487 at commit 4098407.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-05-10T16:15:22Z

Thanks all!

…d to avoid OOM ### What changes were proposed in this pull request? This patch proposes to increase the maximum heap memory setting for release build. ### Why are the changes needed? When I was cutting RCs for 2.4.8, I frequently encountered OOM during building using mvn. It happens many times until I increased the heap memory setting. I am not sure if other release managers encounter the same issue. So I propose to increase the heap memory setting and see if it looks good for others. ### Does this PR introduce _any_ user-facing change? No, dev only. ### How was this patch tested? Manually used it during cutting RCs of 2.4.8. Closes apache#32487 from viirya/release-mvn-oom. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 20d3224) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Set jvm memory for release build.

2d27589

github-actions bot added BUILD INFRA labels May 9, 2021

viirya changed the title ~~[SPARK-35358][BUILD] Increase maximum Java heap used for release build~~ [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM May 9, 2021

dongjoon-hyun reviewed May 9, 2021

View reviewed changes

For review comment.

4098407

HyukjinKwon approved these changes May 10, 2021

View reviewed changes

cloud-fan approved these changes May 10, 2021

View reviewed changes

dongjoon-hyun approved these changes May 10, 2021

View reviewed changes

dongjoon-hyun closed this in 20d3224 May 10, 2021

viirya deleted the release-mvn-oom branch May 10, 2021 16:15

[SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM #32487

[SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM #32487

Uh oh!

Conversation

viirya commented May 9, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented May 9, 2021

Uh oh!

SparkQA commented May 9, 2021

Uh oh!

SparkQA commented May 9, 2021

Uh oh!

dongjoon-hyun May 9, 2021

Choose a reason for hiding this comment

Uh oh!

viirya May 9, 2021

Choose a reason for hiding this comment

Uh oh!

maropu May 10, 2021

Choose a reason for hiding this comment

Uh oh!

viirya May 10, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 9, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

srowen commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

viirya commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

dongjoon-hyun commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

SparkQA commented May 10, 2021

Uh oh!

viirya commented May 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants