Skip to content

Commit f46a074

Browse files
sylvinusrxin
authored andcommitted
[SPARK-16740][SQL] Fix Long overflow in LongToUnsafeRowMap
Avoid overflow of Long type causing a NegativeArraySizeException a few lines later. Unit tests for HashedRelationSuite still pass. I can confirm the python script I included in https://issues.apache.org/jira/browse/SPARK-16740 works fine with this patch. Unfortunately I don't have the knowledge/time to write a Scala test case for HashedRelationSuite right now. As the patch is pretty obvious I hope it can be included without this. Thanks! Author: Sylvain Zimmer <sylvain@sylvainzimmer.com> Closes #14373 from sylvinus/master. (cherry picked from commit 1178d61) Signed-off-by: Reynold Xin <rxin@databricks.com>
1 parent 825c837 commit f46a074

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -608,7 +608,8 @@ private[execution] final class LongToUnsafeRowMap(val mm: TaskMemoryManager, cap
608608
def optimize(): Unit = {
609609
val range = maxKey - minKey
610610
// Convert to dense mode if it does not require more memory or could fit within L1 cache
611-
if (range < array.length || range < 1024) {
611+
// SPARK-16740: Make sure range doesn't overflow if minKey has a large negative value
612+
if (range >= 0 && (range < array.length || range < 1024)) {
612613
try {
613614
ensureAcquireMemory((range + 1) * 8L)
614615
} catch {

0 commit comments

Comments
 (0)