Commit 17a5007
[SPARK-30865][SQL][SS] Refactor DateTimeUtils
### What changes were proposed in this pull request?
1. Move TimeZoneUTC and TimeZoneGMT to DateTimeTestUtils
2. Remove TimeZoneGMT
3. Use ZoneId.systemDefault() instead of defaultTimeZone().toZoneId
4. Alias SQLDate & SQLTimestamp to internal types of DateType and TimestampType
5. Avoid one `*` `DateTimeUtils`.`in fromJulianDay()`
6. Use toTotalMonths in `DateTimeUtils`.`subtractDates()`
7. Remove `julianCommonEraStart`, `timestampToString()`, `microsToEpochDays()`, `epochDaysToMicros()`, `instantToDays()` from `DateTimeUtils`.
8. Make splitDate() private.
9. Remove `def daysToMicros(days: Int): Long` and `def microsToDays(micros: Long): Int`.
### Why are the changes needed?
This simplifies the common code related to date-time operations, and should improve maintainability. In particular:
1. TimeZoneUTC and TimeZoneGMT are moved to DateTimeTestUtils because they are used only in tests
2. TimeZoneGMT can be removed because it is equal to TimeZoneUTC
3. After the PR #27494, Spark expressions and DateTimeUtils functions switched to ZoneId instead of TimeZone completely. `defaultTimeZone()` with `TimeZone` as return type is not needed anymore.
4. SQLDate and SQLTimestamp types can be explicitly aliased to internal types of DateType and and TimestampType instead of declaring this in a comment.
5. Avoid one `*` `DateTimeUtils`.`in fromJulianDay()`.
6. Use toTotalMonths in `DateTimeUtils`.`subtractDates()`.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
By existing test suites
Closes #27617 from MaxGekk/move-time-zone-consts.
Lead-authored-by: Max Gekk <max.gekk@gmail.com>
Co-authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent 8750363 commit 17a5007
File tree
22 files changed
+241
-274
lines changed- sql
- catalyst/src
- main/scala/org/apache/spark/sql/catalyst
- expressions
- util
- test/scala/org/apache/spark/sql
- catalyst
- expressions
- json
- util
- util
- core/src
- main/scala/org/apache/spark/sql/execution/datasources
- jdbc
- parquet
- test/scala/org/apache/spark/sql
- execution/datasources
- json
- parquet
- sources
- streaming
22 files changed
+241
-274
lines changedLines changed: 6 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
303 | 303 | | |
304 | 304 | | |
305 | 305 | | |
306 | | - | |
| 306 | + | |
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
| |||
443 | 443 | | |
444 | 444 | | |
445 | 445 | | |
446 | | - | |
| 446 | + | |
447 | 447 | | |
448 | 448 | | |
449 | 449 | | |
| |||
480 | 480 | | |
481 | 481 | | |
482 | 482 | | |
483 | | - | |
| 483 | + | |
484 | 484 | | |
485 | 485 | | |
486 | 486 | | |
| |||
1034 | 1034 | | |
1035 | 1035 | | |
1036 | 1036 | | |
1037 | | - | |
1038 | | - | |
| 1037 | + | |
1039 | 1038 | | |
1040 | 1039 | | |
1041 | 1040 | | |
| |||
1120 | 1119 | | |
1121 | 1120 | | |
1122 | 1121 | | |
1123 | | - | |
| 1122 | + | |
1124 | 1123 | | |
1125 | 1124 | | |
1126 | 1125 | | |
| |||
1247 | 1246 | | |
1248 | 1247 | | |
1249 | 1248 | | |
1250 | | - | |
| 1249 | + | |
1251 | 1250 | | |
1252 | 1251 | | |
1253 | 1252 | | |
| |||
Lines changed: 6 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
895 | 895 | | |
896 | 896 | | |
897 | 897 | | |
898 | | - | |
| 898 | + | |
899 | 899 | | |
900 | 900 | | |
901 | 901 | | |
| |||
975 | 975 | | |
976 | 976 | | |
977 | 977 | | |
978 | | - | |
| 978 | + | |
979 | 979 | | |
980 | 980 | | |
981 | 981 | | |
| |||
1242 | 1242 | | |
1243 | 1243 | | |
1244 | 1244 | | |
1245 | | - | |
| 1245 | + | |
1246 | 1246 | | |
1247 | 1247 | | |
1248 | | - | |
| 1248 | + | |
1249 | 1249 | | |
1250 | 1250 | | |
1251 | 1251 | | |
| |||
1261 | 1261 | | |
1262 | 1262 | | |
1263 | 1263 | | |
1264 | | - | |
| 1264 | + | |
1265 | 1265 | | |
1266 | 1266 | | |
1267 | | - | |
| 1267 | + | |
1268 | 1268 | | |
1269 | 1269 | | |
1270 | 1270 | | |
| |||
0 commit comments