Skip to content

Commit a53394f

Browse files
MaxGekkcloud-fan
authored andcommitted
[SPARK-31874][SQL] Use FastDateFormat as the legacy fractional formatter
### What changes were proposed in this pull request? 1. Replace `SimpleDateFormat` by `FastDateFormat` as the legacy formatter of `FractionTimestampFormatter`. 2. Optimise `LegacyFastTimestampFormatter` for `java.sql.Timestamp` w/o fractional part. ### Why are the changes needed? 1. By default `HiveResult`.`hiveResultString` retrieves timestamps values as instances of `java.sql.Timestamp`, and uses the legacy parser `SimpleDateFormat` to convert the timestamps to strings. After the fix #28024, the fractional formatter and its companion - legacy formatter `SimpleDateFormat` are created per every value. By switching from `LegacySimpleTimestampFormatter` to `LegacyFastTimestampFormatter`, we can utilize the internal cache of `FastDateFormat`, and avoid parsing the default pattern `yyyy-MM-dd HH:mm:ss`. 2. The second change in the method `def format(ts: Timestamp): String` of `LegacyFastTimestampFormatter` is needed to optimize the formatter for patterns without the fractional part and avoid conversions to microseconds. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing tests in `TimestampFormatter`. Closes #28678 from MaxGekk/fastdateformat-as-legacy-frac-formatter. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 47dc332) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent dc0d12a commit a53394f

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ class FractionTimestampFormatter(zoneId: ZoneId)
121121
TimestampFormatter.defaultPattern,
122122
zoneId,
123123
TimestampFormatter.defaultLocale,
124+
LegacyDateFormats.FAST_DATE_FORMAT,
124125
needVarLengthSecondFraction = false) {
125126

126127
@transient
@@ -203,7 +204,11 @@ class LegacyFastTimestampFormatter(
203204
}
204205

205206
override def format(ts: Timestamp): String = {
206-
format(fromJavaTimestamp(ts))
207+
if (ts.getNanos == 0) {
208+
fastDateFormat.format(ts)
209+
} else {
210+
format(fromJavaTimestamp(ts))
211+
}
207212
}
208213

209214
override def format(instant: Instant): String = {

0 commit comments

Comments
 (0)