Skip to content

Commit 4cfce3e

Browse files
rahulsmahadevsrowen
authored andcommitted
[SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting string to timestamp
### What changes were proposed in this pull request? * Adding an additional check in `stringToTimestamp` to handle cases where the input has trailing ':' * Added a test to make sure this works. ### Why are the changes needed? In a couple of scenarios while converting from String to Timestamp `DateTimeUtils.stringToTimestamp` throws an array out of bounds exception if there is trailing ':'. The behavior of this method requires it to return `None` in case the format of the string is incorrect. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Added a test in the `DateTimeTestUtils` suite to test if my fix works. Closes #26143 from rahulsmahadev/SPARK-29494. Lead-authored-by: Rahul Mahadev <rahul.mahadev@databricks.com> Co-authored-by: Rahul Shivu Mahadev <51690557+rahulsmahadev@users.noreply.github.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
1 parent 23f45f1 commit 4cfce3e

File tree

2 files changed

+7
-1
lines changed

2 files changed

+7
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,7 @@ object DateTimeUtils {
286286
i += 1
287287
}
288288
} else {
289-
if (b == ':' || b == ' ') {
289+
if (i < segments.length && (b == ':' || b == ' ')) {
290290
segments(i) = currentSegmentValue
291291
currentSegmentValue = 0
292292
i += 1

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -456,6 +456,12 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers {
456456
}
457457
}
458458

459+
test("trailing characters while converting string to timestamp") {
460+
val s = UTF8String.fromString("2019-10-31T10:59:23Z:::")
461+
val time = DateTimeUtils.stringToTimestamp(s, defaultZoneId)
462+
assert(time == None)
463+
}
464+
459465
test("truncTimestamp") {
460466
def testTrunc(
461467
level: Int,

0 commit comments

Comments
 (0)