Skip to content

Commit 403c8d5

Browse files
wangjiaochundongjoon-hyun
authored andcommitted
[SPARK-26287][CORE] Don't need to create an empty spill file when memory has no records
## What changes were proposed in this pull request? If there are no records in memory, then we don't need to create an empty temp spill file. ## How was this patch tested? Existing tests (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Closes apache#23225 from wangjiaochun/ShufflSorter. Authored-by: 10087686 <wang.jiaochun@zte.com.cn> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent ec506bd commit 403c8d5

File tree

2 files changed

+10
-4
lines changed

2 files changed

+10
-4
lines changed

core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,15 @@ final class ShuffleExternalSorter extends MemoryConsumer {
145145
*/
146146
private void writeSortedFile(boolean isLastFile) {
147147

148+
// This call performs the actual sort.
149+
final ShuffleInMemorySorter.ShuffleSorterIterator sortedRecords =
150+
inMemSorter.getSortedIterator();
151+
152+
// If there are no sorted records, so we don't need to create an empty spill file.
153+
if (!sortedRecords.hasNext()) {
154+
return;
155+
}
156+
148157
final ShuffleWriteMetricsReporter writeMetricsToUse;
149158

150159
if (isLastFile) {
@@ -157,10 +166,6 @@ private void writeSortedFile(boolean isLastFile) {
157166
writeMetricsToUse = new ShuffleWriteMetrics();
158167
}
159168

160-
// This call performs the actual sort.
161-
final ShuffleInMemorySorter.ShuffleSorterIterator sortedRecords =
162-
inMemSorter.getSortedIterator();
163-
164169
// Small writes to DiskBlockObjectWriter will be fairly inefficient. Since there doesn't seem to
165170
// be an API to directly transfer bytes from managed memory to the disk writer, we buffer
166171
// data through a byte array. This array does not need to be large enough to hold a single

core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,7 @@ public void writeEmptyIterator() throws Exception {
235235
final Option<MapStatus> mapStatus = writer.stop(true);
236236
assertTrue(mapStatus.isDefined());
237237
assertTrue(mergedOutputFile.exists());
238+
assertEquals(0, spillFilesCreated.size());
238239
assertArrayEquals(new long[NUM_PARTITITONS], partitionSizesInMergedFile);
239240
assertEquals(0, taskMetrics.shuffleWriteMetrics().recordsWritten());
240241
assertEquals(0, taskMetrics.shuffleWriteMetrics().bytesWritten());

0 commit comments

Comments
 (0)