Skip to content

Commit 5903dab

Browse files
bchochorxin
authored andcommitted
[SPARK-16827][BRANCH-2.0] Avoid reporting spill metrics as shuffle metrics
## What changes were proposed in this pull request? Fix a bug where spill metrics were being reported as shuffle metrics. Eventually these spill metrics should be reported (SPARK-3577), but separate from shuffle metrics. The fix itself basically reverts the line to what it was in 1.6. ## How was this patch tested? Cherry-picked from master (#15347) Author: Brian Cho <bcho@fb.com> Closes #15455 from dafrista/shuffle-metrics-2.0.
1 parent 050b817 commit 5903dab

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,9 @@ private UnsafeExternalSorter(
144144
// Use getSizeAsKb (not bytes) to maintain backwards compatibility for units
145145
// this.fileBufferSizeBytes = (int) conf.getSizeAsKb("spark.shuffle.file.buffer", "32k") * 1024;
146146
this.fileBufferSizeBytes = 32 * 1024;
147-
this.writeMetrics = taskContext.taskMetrics().shuffleWriteMetrics();
147+
// The spill metrics are stored in a new ShuffleWriteMetrics, and then discarded (this fixes SPARK-16827).
148+
// TODO: Instead, separate spill metrics should be stored and reported (tracked in SPARK-3577).
149+
this.writeMetrics = new ShuffleWriteMetrics();
148150

149151
if (existingInMemorySorter == null) {
150152
this.inMemSorter = new UnsafeInMemorySorter(

0 commit comments

Comments
 (0)