Commit 500877e
[SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence
### What changes were proposed in this pull request?
1.Add time field steps check for date start/end in Sequence at `org.apache.spark.sql.catalyst.expressions.Sequence.TemporalSequenceImpl`
2.Add a UT:`SPARK-32133: Sequence step must be a day interval if start and end values are dates` at `org.apache.spark.sql.catalyst.expressions.CollectionExpressionsSuite`
### Why are the changes needed?
**Sequence time field steps for date start/end looks strange in spark as follows:**
```
scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 hour))").head(3)
res0: Array[org.apache.spark.sql.Row] = _Array([2011-03-01], [2011-03-01], [2011-03-01])_ **<- strange result.**
scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 day))").head(3)
res1: Array[org.apache.spark.sql.Row] = Array([2011-03-01], [2011-03-02])
```
**While this behavior in Prosto make sense:**
```
presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' hour);
Query 20200624_122744_00002_pehix failed: sequence step must be a day interval if start and end values are dates
presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' day);
_col0
[2011-03-01, 2011-03-02]
```
### Does this PR introduce _any_ user-facing change?
Yes, after this patch, users will get informed `sequence step must be a day interval if start and end values are dates` when
use time field steps for date start/end in Sequence.
### How was this patch tested?
Unit test.
Closes #28926 from TJX2014/master-SPARK-31982-sequence-cross-dst-follow-presto.
Authored-by: TJX2014 <xiaoxingstack@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>1 parent 560fe1f commit 500877e
File tree
2 files changed
+27
-0
lines changed- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
2 files changed
+27
-0
lines changedLines changed: 20 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2612 | 2612 | | |
2613 | 2613 | | |
2614 | 2614 | | |
| 2615 | + | |
| 2616 | + | |
| 2617 | + | |
| 2618 | + | |
| 2619 | + | |
2615 | 2620 | | |
| 2621 | + | |
2616 | 2622 | | |
2617 | 2623 | | |
2618 | 2624 | | |
| 2625 | + | |
2619 | 2626 | | |
2620 | 2627 | | |
2621 | 2628 | | |
| |||
2674 | 2681 | | |
2675 | 2682 | | |
2676 | 2683 | | |
| 2684 | + | |
| 2685 | + | |
| 2686 | + | |
| 2687 | + | |
| 2688 | + | |
| 2689 | + | |
| 2690 | + | |
| 2691 | + | |
| 2692 | + | |
| 2693 | + | |
| 2694 | + | |
2677 | 2695 | | |
2678 | 2696 | | |
2679 | 2697 | | |
2680 | 2698 | | |
2681 | 2699 | | |
| 2700 | + | |
| 2701 | + | |
2682 | 2702 | | |
2683 | 2703 | | |
2684 | 2704 | | |
| |||
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
933 | 933 | | |
934 | 934 | | |
935 | 935 | | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
936 | 943 | | |
937 | 944 | | |
938 | 945 | | |
| |||
0 commit comments