Skip to content

Commit

Permalink
rdar://100981691 Set spark.cleaner.periodicGC.interval to 20min
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

We used this in Spark 3.3 since 2022-10-17 (3.3.0.22-apple). This is a backport from Spark 3.3 to 3.2.
(cherry picked from commit fbd7496)

### Why are the changes needed?

This is used commonly in the production streaming job memory optimization.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.
  • Loading branch information
dongjoon-hyun authored and GitHub Enterprise committed Nov 7, 2022
1 parent 8d19fb5 commit 4e48e5e
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1649,7 +1649,7 @@ package object config {
ConfigBuilder("spark.cleaner.periodicGC.interval")
.version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("30min")
.createWithDefaultString("20min")

private[spark] val CLEANER_REFERENCE_TRACKING =
ConfigBuilder("spark.cleaner.referenceTracking")
Expand Down

0 comments on commit 4e48e5e

Please sign in to comment.