Skip to content

[SPARK-55714][SQL] JDK might throw ArithmeticException without message #54514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pan3793 wants to merge 1 commit into
base: master
Choose a base branch
from
+61 −52
Open

[SPARK-55714][SQL] JDK might throw ArithmeticException without message #54514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion sql/api/src/main/scala/org/apache/spark/sql/errors/ExecutionErrors.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,22 @@ private[sql] trait ExecutionErrors extends DataTypeErrorsBase {
message: String,
suggestedFunc: String = "",
context: QueryContext = null): ArithmeticException = {
val canonicalMessage = message match {
// For "hot throws", the JIT will produce compiled code that does not
// go through the interpreter via deoptimization to throw an exception
// but throws a pre-allocated exception object without a stack trace
// or message. See https://bugs.openjdk.org/browse/JDK-8367990
case null => "overflow"
case m if m.contains("overflow") => "overflow"
case m => m
}
Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a simple Scala-style one-liner?

-    val canonicalMessage = message match {
-      // For "hot throws", the JIT will produce compiled code that does not
-      // go through the interpreter via deoptimization to throw an exception
-      // but throws a pre-allocated exception object without a stack trace
-      // or message. See https://bugs.openjdk.org/browse/JDK-8367990
-      case null => "overflow"
-      case m if m.contains("overflow") => "overflow"
-      case m => m
-    }
+    val canonicalMessage = Option(message).filterNot(_.contains("overflow")).getOrElse("overflow")

Copy link
Member Author

@pan3793 pan3793 Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, they are semantic equals, but the one-line version takes me a few seconds to understand (maybe I'm not good at functional programming). I prefer the current pattern match style for better readability.

val alternative = if (suggestedFunc.nonEmpty) {
s" Use '$suggestedFunc' to tolerate overflow and return NULL instead."
} else ""
new SparkArithmeticException(
errorClass = "ARITHMETIC_OVERFLOW",
messageParameters = Map(
"message" -> message,
"message" -> canonicalMessage,
"alternative" -> alternative,
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY)),
context = getQueryContext(context),
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,7 @@ class CatalystTypeConvertersSuite extends SparkFunSuite with SQLHelper {
val errMsg = intercept[ArithmeticException] {
IntervalUtils.durationToMicros(Duration.ofSeconds(Long.MaxValue, Long.MaxValue))
}.getMessage
assert(errMsg.contains("long overflow"))
assert(errMsg == null || errMsg.contains("overflow"))
}

test("SPARK-35726: Truncate java.time.Duration by fields of day-time interval type") {
Expand Down Expand Up @@ -357,7 +357,7 @@ class CatalystTypeConvertersSuite extends SparkFunSuite with SQLHelper {
val errMsg = intercept[ArithmeticException] {
IntervalUtils.periodToMonths(Period.of(Int.MaxValue, Int.MaxValue, Int.MaxValue))
}.getMessage
assert(errMsg.contains("integer overflow"))
assert(errMsg == null || errMsg.contains("overflow"))
}

test("SPARK-35769: Truncate java.time.Period by fields of year-month interval type") {
Expand Down
8 changes: 4 additions & 4 deletions .../src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,19 +119,19 @@ class ArithmeticExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
checkErrorInExpression[SparkArithmeticException](
UnaryMinus(Literal(Long.MinValue)), "ARITHMETIC_OVERFLOW",
Map("message" -> "long overflow", "alternative" -> "",
Map("message" -> "overflow", "alternative" -> "",
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY)))
checkErrorInExpression[SparkArithmeticException](
UnaryMinus(Literal(Int.MinValue)), "ARITHMETIC_OVERFLOW",
Map("message" -> "integer overflow", "alternative" -> "",
Map("message" -> "overflow", "alternative" -> "",
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY)))
checkErrorInExpression[SparkArithmeticException](
UnaryMinus(Literal(Short.MinValue)), "ARITHMETIC_OVERFLOW",
Map("message" -> "short overflow", "alternative" -> "",
Map("message" -> "overflow", "alternative" -> "",
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY)))
checkErrorInExpression[SparkArithmeticException](
UnaryMinus(Literal(Byte.MinValue)), "ARITHMETIC_OVERFLOW",
Map("message" -> "byte overflow", "alternative" -> "",
Map("message" -> "overflow", "alternative" -> "",
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY)))
checkEvaluation(UnaryMinus(positiveShortLit), (- positiveShort).toShort)
checkEvaluation(UnaryMinus(negativeShortLit), (- negativeShort).toShort)
Expand Down
6 changes: 3 additions & 3 deletions ...alyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1307,7 +1307,7 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
timeZoneId = Some(tz)),
Duration.ZERO)
}.getMessage
assert(errMsg.contains("overflow"))
assert(errMsg == null || errMsg.contains("overflow"))

Seq(false, true).foreach { legacy =>
checkConsistencyBetweenInterpretedAndCodegen(
Expand Down Expand Up @@ -1372,7 +1372,7 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
timeZoneId = Some(tz)),
Duration.ZERO)
}.getMessage
assert(errMsg.contains("overflow"))
assert(errMsg == null || errMsg.contains("overflow"))

Seq(false, true).foreach { legacy =>
checkConsistencyBetweenInterpretedAndCodegen(
Expand Down Expand Up @@ -1822,7 +1822,7 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
null)
}.getCause
assert(e.isInstanceOf[ArithmeticException])
assert(e.getMessage.contains("long overflow"))
assert(e.getMessage == null || e.getMessage.contains("overflow"))

checkEvaluation(
TimestampAddInterval(
Expand Down
4 changes: 2 additions & 2 deletions ...t/src/test/scala/org/apache/spark/sql/catalyst/expressions/IntervalExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ class IntervalExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
check("-100 years -1 millisecond", 0.5, "-50 years -500 microseconds")
check("2 months 4 seconds", -0.5, "-1 months -2 seconds")
check("1 month 2 microseconds", 1.5, "1 months 3 microseconds")
check("2 months", Int.MaxValue, "integer overflow", Some(true))
check("2 months", Int.MaxValue, "overflow", Some(true))
check("2 months", Int.MaxValue, s"${Int.MaxValue} months", Some(false))
}

Expand Down Expand Up @@ -188,7 +188,7 @@ class IntervalExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
check("1 month 3 microsecond", 1.5, "2 microseconds")
check("1 second", 0, "Division by zero", Some(true))
check("1 second", 0, null, Some(false))
check(s"${Int.MaxValue} months", 0.9, "integer overflow", Some(true))
check(s"${Int.MaxValue} months", 0.9, "overflow", Some(true))
check(s"${Int.MaxValue} months", 0.9, s"${Int.MaxValue} months", Some(false))
}

Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -832,7 +832,7 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
val msg = intercept[ArithmeticException] {
DateTimeUtils.localDateTimeToMicros(dt)
}.getMessage
assert(msg == "long overflow")
assert(msg == null || msg.contains("overflow"))
}
}

Expand Down Expand Up @@ -1445,7 +1445,7 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
},
condition = "ARITHMETIC_OVERFLOW",
parameters = Map(
"message" -> "long overflow",
"message" -> "overflow",
"alternative" -> "",
"config" -> toSQLConf(SqlApiConf.ANSI_ENABLED_KEY))
)
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
assert(divide(interval, 0.9) === new CalendarInterval(Int.MaxValue, Int.MaxValue,
((Int.MaxValue / 9.0) * MICROS_PER_DAY).round))
val e1 = intercept[ArithmeticException](divideExact(interval, 0.9))
assert(e1.getMessage.contains("integer overflow"))
assert(e1.getMessage.contains("overflow"))

interval = new CalendarInterval(123, 456, 789)
assert(divide(interval, 0) === null)
Expand Down
12 changes: 6 additions & 6 deletions sql/core/src/test/resources/sql-tests/results/postgreSQL/int4.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -238,7 +238,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -273,7 +273,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -309,7 +309,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -345,7 +345,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_subtract' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -381,7 +381,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_subtract' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
8 changes: 4 additions & 4 deletions sql/core/src/test/resources/sql-tests/results/postgreSQL/int8.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -918,7 +918,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -958,7 +958,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -998,7 +998,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
8 changes: 4 additions & 4 deletions sql/core/src/test/resources/sql-tests/results/postgreSQL/int8.sql.out.java21
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -918,7 +918,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -958,7 +958,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -998,7 +998,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_multiply' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/sql-tests/results/postgreSQL/window_part2.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand All @@ -251,7 +251,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/sql-udf.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1111,7 +1111,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/sql-tests/results/try_aggregates.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -365,7 +365,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/sql-tests/results/try_aggregates.sql.out.java21
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -365,7 +365,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
16 changes: 8 additions & 8 deletions sql/core/src/test/resources/sql-tests/results/try_arithmetic.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand All @@ -91,7 +91,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -307,7 +307,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand All @@ -331,7 +331,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -505,7 +505,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand All @@ -529,7 +529,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down Expand Up @@ -671,7 +671,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "integer overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand All @@ -695,7 +695,7 @@ org.apache.spark.SparkArithmeticException
"messageParameters" : {
"alternative" : " Use 'try_add' to tolerate overflow and return NULL instead.",
"config" : "\"spark.sql.ansi.enabled\"",
"message" : "long overflow"
"message" : "overflow"
},
"queryContext" : [ {
"objectType" : "",
Expand Down
Loading