Skip to content

Commit be4faaf

Browse files
cloud-fangatorsmile
cloud-fan
authored and
gatorsmile
committed
Revert "[SPARK-23264][SQL] Make INTERVAL keyword optional when ANSI enabled"
### What changes were proposed in this pull request? Revert #20433 . ### Why are the changes needed? According to the SQL standard, the INTERVAL prefix is required: ``` <interval literal> ::= INTERVAL [ <sign> ] <interval string> <interval qualifier> <interval string> ::= <quote> <unquoted interval string> <quote> ``` ### Does this PR introduce any user-facing change? yes, but omitting the INTERVAL prefix is a new feature in 3.0 ### How was this patch tested? existing tests Closes #27080 from cloud-fan/interval. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Xiao Li <gatorsmile@gmail.com>
1 parent e645125 commit be4faaf

File tree

7 files changed

+16
-288
lines changed

7 files changed

+16
-288
lines changed

‎docs/sql-keywords.md

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ license: |
2222
When `spark.sql.ansi.enabled` is true, Spark SQL will use the ANSI mode parser.
2323
In this mode, Spark SQL has two kinds of keywords:
2424
* Reserved keywords: Keywords that are reserved and can't be used as identifiers for table, view, column, function, alias, etc.
25-
* Non-reserved keywords: Keywords that have a special meaning only in particular contexts and can be used as identifiers in other contexts. For example, `SELECT 1 WEEK` is an interval literal, but WEEK can be used as identifiers in other places.
25+
* Non-reserved keywords: Keywords that have a special meaning only in particular contexts and can be used as identifiers in other contexts. For example, `EXPLAIN SELECT ...` is a command, but EXPLAIN can be used as identifiers in other places.
2626

2727
When the ANSI mode is disabled, Spark SQL has two kinds of keywords:
2828
* Non-reserved keywords: Same definition as the one when the ANSI mode enabled.
@@ -88,7 +88,6 @@ Below is a list of all the keywords in Spark SQL.
8888
<tr><td>DATABASE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
8989
<tr><td>DATABASES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
9090
<tr><td>DAY</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
91-
<tr><td>DAYS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
9291
<tr><td>DBPROPERTIES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
9392
<tr><td>DEFINED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
9493
<tr><td>DELETE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
@@ -136,7 +135,6 @@ Below is a list of all the keywords in Spark SQL.
136135
<tr><td>GROUPING</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
137136
<tr><td>HAVING</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
138137
<tr><td>HOUR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
139-
<tr><td>HOURS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
140138
<tr><td>IF</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
141139
<tr><td>IGNORE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
142140
<tr><td>IMPORT</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
@@ -174,15 +172,9 @@ Below is a list of all the keywords in Spark SQL.
174172
<tr><td>MAP</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
175173
<tr><td>MATCHED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
176174
<tr><td>MERGE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
177-
<tr><td>MICROSECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
178-
<tr><td>MICROSECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
179-
<tr><td>MILLISECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
180-
<tr><td>MILLISECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
181175
<tr><td>MINUS</td><td>reserved</td><td>strict-non-reserved</td><td>non-reserved</td></tr>
182176
<tr><td>MINUTE</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
183-
<tr><td>MINUTES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
184177
<tr><td>MONTH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
185-
<tr><td>MONTHS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
186178
<tr><td>MSCK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
187179
<tr><td>NAMESPACE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
188180
<tr><td>NAMESPACES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
@@ -242,7 +234,6 @@ Below is a list of all the keywords in Spark SQL.
242234
<tr><td>ROWS</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
243235
<tr><td>SCHEMA</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
244236
<tr><td>SECOND</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
245-
<tr><td>SECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
246237
<tr><td>SELECT</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
247238
<tr><td>SEMI</td><td>reserved</td><td>strict-non-reserved</td><td>non-reserved</td></tr>
248239
<tr><td>SEPARATED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
@@ -293,12 +284,9 @@ Below is a list of all the keywords in Spark SQL.
293284
<tr><td>USING</td><td>reserved</td><td>strict-non-reserved</td><td>reserved</td></tr>
294285
<tr><td>VALUES</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
295286
<tr><td>VIEW</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
296-
<tr><td>WEEK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
297-
<tr><td>WEEKS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
298287
<tr><td>WHEN</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
299288
<tr><td>WHERE</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
300289
<tr><td>WINDOW</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
301290
<tr><td>WITH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
302291
<tr><td>YEAR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
303-
<tr><td>YEARS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
304292
</table>

‎sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

Lines changed: 2 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -816,7 +816,6 @@ booleanValue
816816

817817
interval
818818
: INTERVAL (errorCapturingMultiUnitsInterval | errorCapturingUnitToUnitInterval)?
819-
| {SQL_standard_keyword_behavior}? (errorCapturingMultiUnitsInterval | errorCapturingUnitToUnitInterval)
820819
;
821820

822821
errorCapturingMultiUnitsInterval
@@ -842,23 +841,12 @@ intervalValue
842841

843842
intervalUnit
844843
: DAY
845-
| DAYS
846844
| HOUR
847-
| HOURS
848-
| MICROSECOND
849-
| MICROSECONDS
850-
| MILLISECOND
851-
| MILLISECONDS
852845
| MINUTE
853-
| MINUTES
854846
| MONTH
855-
| MONTHS
856847
| SECOND
857-
| SECONDS
858-
| WEEK
859-
| WEEKS
860848
| YEAR
861-
| YEARS
849+
| identifier
862850
;
863851

864852
colPosition
@@ -994,7 +982,7 @@ number
994982
// function, alias, etc.
995983
// - Non-reserved keywords:
996984
// Keywords that have a special meaning only in particular contexts and can be used as
997-
// identifiers in other contexts. For example, `SELECT 1 WEEK` is an interval literal, but WEEK
985+
// identifiers in other contexts. For example, `EXPLAIN SELECT ...` is a command, but EXPLAIN
998986
// can be used as identifiers in other places.
999987
// You can find the full keywords list by searching "Start of the keywords list" in this file.
1000988
// The non-reserved keywords are listed below. Keywords not in this list are reserved keywords.
@@ -1032,7 +1020,6 @@ ansiNonReserved
10321020
| DATA
10331021
| DATABASE
10341022
| DATABASES
1035-
| DAYS
10361023
| DBPROPERTIES
10371024
| DEFINED
10381025
| DELETE
@@ -1063,7 +1050,6 @@ ansiNonReserved
10631050
| FUNCTIONS
10641051
| GLOBAL
10651052
| GROUPING
1066-
| HOURS
10671053
| IF
10681054
| IGNORE
10691055
| IMPORT
@@ -1092,12 +1078,6 @@ ansiNonReserved
10921078
| MAP
10931079
| MATCHED
10941080
| MERGE
1095-
| MICROSECOND
1096-
| MICROSECONDS
1097-
| MILLISECOND
1098-
| MILLISECONDS
1099-
| MINUTES
1100-
| MONTHS
11011081
| MSCK
11021082
| NAMESPACE
11031083
| NAMESPACES
@@ -1144,7 +1124,6 @@ ansiNonReserved
11441124
| ROW
11451125
| ROWS
11461126
| SCHEMA
1147-
| SECONDS
11481127
| SEPARATED
11491128
| SERDE
11501129
| SERDEPROPERTIES
@@ -1182,10 +1161,7 @@ ansiNonReserved
11821161
| USE
11831162
| VALUES
11841163
| VIEW
1185-
| WEEK
1186-
| WEEKS
11871164
| WINDOW
1188-
| YEARS
11891165
;
11901166

11911167
// When `SQL_standard_keyword_behavior=false`, there are 2 kinds of keywords in Spark SQL.
@@ -1267,7 +1243,6 @@ nonReserved
12671243
| DATABASE
12681244
| DATABASES
12691245
| DAY
1270-
| DAYS
12711246
| DBPROPERTIES
12721247
| DEFINED
12731248
| DELETE
@@ -1313,7 +1288,6 @@ nonReserved
13131288
| GROUPING
13141289
| HAVING
13151290
| HOUR
1316-
| HOURS
13171291
| IF
13181292
| IGNORE
13191293
| IMPORT
@@ -1347,14 +1321,8 @@ nonReserved
13471321
| MAP
13481322
| MATCHED
13491323
| MERGE
1350-
| MICROSECOND
1351-
| MICROSECONDS
1352-
| MILLISECOND
1353-
| MILLISECONDS
13541324
| MINUTE
1355-
| MINUTES
13561325
| MONTH
1357-
| MONTHS
13581326
| MSCK
13591327
| NAMESPACE
13601328
| NAMESPACES
@@ -1411,7 +1379,6 @@ nonReserved
14111379
| ROWS
14121380
| SCHEMA
14131381
| SECOND
1414-
| SECONDS
14151382
| SELECT
14161383
| SEPARATED
14171384
| SERDE
@@ -1460,14 +1427,11 @@ nonReserved
14601427
| USER
14611428
| VALUES
14621429
| VIEW
1463-
| WEEK
1464-
| WEEKS
14651430
| WHEN
14661431
| WHERE
14671432
| WINDOW
14681433
| WITH
14691434
| YEAR
1470-
| YEARS
14711435
;
14721436

14731437
// NOTE: If you add a new token in the list below, you should update the list of keywords
@@ -1530,7 +1494,6 @@ DATA: 'DATA';
15301494
DATABASE: 'DATABASE';
15311495
DATABASES: 'DATABASES' | 'SCHEMAS';
15321496
DAY: 'DAY';
1533-
DAYS: 'DAYS';
15341497
DBPROPERTIES: 'DBPROPERTIES';
15351498
DEFINED: 'DEFINED';
15361499
DELETE: 'DELETE';
@@ -1577,7 +1540,6 @@ GROUP: 'GROUP';
15771540
GROUPING: 'GROUPING';
15781541
HAVING: 'HAVING';
15791542
HOUR: 'HOUR';
1580-
HOURS: 'HOURS';
15811543
IF: 'IF';
15821544
IGNORE: 'IGNORE';
15831545
IMPORT: 'IMPORT';
@@ -1615,14 +1577,8 @@ MACRO: 'MACRO';
16151577
MAP: 'MAP';
16161578
MATCHED: 'MATCHED';
16171579
MERGE: 'MERGE';
1618-
MICROSECOND: 'MICROSECOND';
1619-
MICROSECONDS: 'MICROSECONDS';
1620-
MILLISECOND: 'MILLISECOND';
1621-
MILLISECONDS: 'MILLISECONDS';
16221580
MINUTE: 'MINUTE';
1623-
MINUTES: 'MINUTES';
16241581
MONTH: 'MONTH';
1625-
MONTHS: 'MONTHS';
16261582
MSCK: 'MSCK';
16271583
NAMESPACE: 'NAMESPACE';
16281584
NAMESPACES: 'NAMESPACES';
@@ -1682,7 +1638,6 @@ ROW: 'ROW';
16821638
ROWS: 'ROWS';
16831639
SCHEMA: 'SCHEMA';
16841640
SECOND: 'SECOND';
1685-
SECONDS: 'SECONDS';
16861641
SELECT: 'SELECT';
16871642
SEMI: 'SEMI';
16881643
SEPARATED: 'SEPARATED';
@@ -1735,14 +1690,11 @@ USER: 'USER';
17351690
USING: 'USING';
17361691
VALUES: 'VALUES';
17371692
VIEW: 'VIEW';
1738-
WEEK: 'WEEK';
1739-
WEEKS: 'WEEKS';
17401693
WHEN: 'WHEN';
17411694
WHERE: 'WHERE';
17421695
WINDOW: 'WINDOW';
17431696
WITH: 'WITH';
17441697
YEAR: 'YEAR';
1745-
YEARS: 'YEARS';
17461698
//============================
17471699
// End of the keywords list
17481700
//============================

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala

Lines changed: 1 addition & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -645,11 +645,6 @@ class ExpressionParserSuite extends AnalysisTest {
645645
"-" -> UnaryMinus(expected)
646646
).foreach { case (sign, expectedLiteral) =>
647647
assertEqual(s"${sign}interval $intervalValue", expectedLiteral)
648-
649-
// SPARK-23264 Support interval values without INTERVAL clauses if ANSI SQL enabled
650-
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
651-
assertEqual(intervalValue, expected)
652-
}
653648
}
654649
}
655650

@@ -687,8 +682,7 @@ class ExpressionParserSuite extends AnalysisTest {
687682
Literal(IntervalUtils.stringToInterval("1 second 1 millisecond")))
688683

689684
// Non Existing unit
690-
intercept("interval 10 nanoseconds",
691-
"no viable alternative at input '10 nanoseconds'")
685+
intercept("interval 10 nanoseconds", "invalid unit 'nanoseconds'")
692686

693687
// Year-Month intervals.
694688
val yearMonthValues = Seq("123-10", "496-0", "-2-3", "-123-0")
@@ -732,34 +726,6 @@ class ExpressionParserSuite extends AnalysisTest {
732726
Literal(new CalendarInterval(3, 4, 22001000L)))
733727
}
734728

735-
test("SPARK-23264 Interval Compatibility tests") {
736-
def checkIntervals(intervalValue: String, expected: Literal): Unit = {
737-
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
738-
assertEqual(intervalValue, expected)
739-
}
740-
741-
// Compatibility tests: If ANSI SQL disabled, `intervalValue` should be parsed as an alias
742-
withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
743-
val aliases = defaultParser.parseExpression(intervalValue).collect {
744-
case a @ Alias(_: Literal, name)
745-
if intervalUnits.exists { unit => name.startsWith(unit.toString) } => a
746-
}
747-
assert(aliases.size === 1)
748-
}
749-
}
750-
val forms = Seq("", "s")
751-
val values = Seq("5", "1", "-11", "8")
752-
intervalUnits.foreach { unit =>
753-
forms.foreach { form =>
754-
values.foreach { value =>
755-
val expected = intervalLiteral(unit, value)
756-
checkIntervals(s"$value $unit$form", expected)
757-
checkIntervals(s"'$value' $unit$form", expected)
758-
}
759-
}
760-
}
761-
}
762-
763729
test("composed expressions") {
764730
assertEqual("1 + r.r As q", (Literal(1) + UnresolvedAttribute("r.r")).as("q"))
765731
assertEqual("1 - f('o', o(bar))", Literal(1) - 'f.function("o", 'o.function('bar)))

0 commit comments

Comments
 (0)