Commit 1af7072
[SPARK-36970][SQL] Manual disabled format
### What changes were proposed in this pull request?
The `date_format` function with `B` format has different behavior when use Java 8 and Java 17, `select date_format('2018-11-17 13:33:33.333', 'B')` in `datetime-formatting-invalid.sql` can prove this.
The case result with Java 8 is
```
-- !query
select date_format('2018-11-17 13:33:33.333', 'B')
-- !query schema
struct<>
-- !query output
java.lang.IllegalArgumentException
Unknown pattern letter: B
```
and the case result with Java 17 is
```
- datetime-formatting-invalid.sql *** FAILED ***
datetime-formatting-invalid.sql
Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
select date_format('2018-11-17 13:33:33.333', 'B')
-- !query schema
struct<date_format(2018-11-17 13:33:33.333, B):string>
-- !query output
in the afternoon (SQLQueryTestSuite.scala:469)
```
We found that this is due to the new support of format `B` in Java 17
```
'B' is used to represent Pattern letters to output a day period in Java 17
* Pattern Count Equivalent builder methods
* ------- ----- --------------------------
* B 1 appendDayPeriodText(TextStyle.SHORT)
* BBBB 4 appendDayPeriodText(TextStyle.FULL)
* BBBBB 5 appendDayPeriodText(TextStyle.NARROW)
```
And through [ http://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html]( http://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html) , we can confirm that format `B` is not documented/supported for `date_format` function currently.
So the main change of this pr is manual disabled format `B` of `date_format` function in `DateTimeFormatterHelper` to make Java 17 compatible with Java 8.
### Why are the changes needed?
Ensure that Java 17 and Java 8 have the same behavior.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test `SQLQueryTestSuite` with JDK 17
**Before**
```
- datetime-formatting-invalid.sql *** FAILED ***
datetime-formatting-invalid.sql
Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
select date_format('2018-11-17 13:33:33.333', 'B')
-- !query schema
struct<date_format(2018-11-17 13:33:33.333, B):string>
-- !query output
in the afternoon (SQLQueryTestSuite.scala:469)
```
**After**
The test `select date_format('2018-11-17 13:33:33.333', 'B')` in `datetime-formatting-invalid.sql` passed
Closes #34237 from LuciferYang/SPARK-36970.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>B of date_format function to make Java 17 compatible with Java 81 parent dc1db95 commit 1af7072
File tree
3 files changed
+7
-3
lines changed- sql
- catalyst/src
- main/scala/org/apache/spark/sql/catalyst/util
- test/scala/org/apache/spark/sql/catalyst/util
- core/src/test/resources/sql-tests/results
3 files changed
+7
-3
lines changedLines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
279 | 279 | | |
280 | 280 | | |
281 | 281 | | |
282 | | - | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
283 | 287 | | |
284 | 288 | | |
285 | 289 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
| 79 | + | |
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
| 317 | + | |
318 | 318 | | |
319 | 319 | | |
320 | 320 | | |
| |||
0 commit comments