[SPARK-4467] Partial fix for fetch failure in sort-based shuffle (1.1)

Andrew Or · Andrew Or · commit aa9ebdaa28be · 2014-11-17T18:10:49.000-08:00
This is the 1.1 version of #3302. There has been some refactoring in master so we can't cherry-pick that PR. Author: Andrew Or <andrew@databricks.com> Closes #3330 from andrewor14/sort-fetch-fail and squashes the following commits: 486fc49 [Andrew Or] Reset `elementsRead`
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
@@ -288,6 +288,7 @@ private[spark] class ExternalSorter[K, V, C](
     myMemoryThreshold = 0
 
     _memoryBytesSpilled += memorySize
+    elementsRead = 0
   }
 
   /**

Original file line number	Diff line number	Diff line change
`@@ -288,6 +288,7 @@ private[spark] class ExternalSorter[K, V, C](`
`288`	`288`	`myMemoryThreshold = 0`
`289`	`289`
`290`	`290`	`_memoryBytesSpilled += memorySize`
	`291`	`+ elementsRead = 0`
`291`	`292`	`}`
`292`	`293`
`293`	`294`	`/**`