Skip to content

Revert "[SPARK-23264][SQL] Make INTERVAL keyword optional when ANSI enabled" #27080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Revert "[SPARK-23264][SQL] Make INTERVAL keyword optional when ANSI enabled" #27080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ee86c8a
make INTERVAL non-optional
cloud-fan Jan 2, 2020
87289cb
fix tests
cloud-fan Jan 3, 2020
91a0e33
still make keywork reserved
cloud-fan Jan 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 1 addition & 13 deletions docs/sql-keywords.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ license: |
When `spark.sql.ansi.enabled` is true, Spark SQL will use the ANSI mode parser.
In this mode, Spark SQL has two kinds of keywords:
* Reserved keywords: Keywords that are reserved and can't be used as identifiers for table, view, column, function, alias, etc.
* Non-reserved keywords: Keywords that have a special meaning only in particular contexts and can be used as identifiers in other contexts. For example, `SELECT 1 WEEK` is an interval literal, but WEEK can be used as identifiers in other places.
* Non-reserved keywords: Keywords that have a special meaning only in particular contexts and can be used as identifiers in other contexts. For example, `EXPLAIN SELECT ...` is a command, but EXPLAIN can be used as identifiers in other places.

When the ANSI mode is disabled, Spark SQL has two kinds of keywords:
* Non-reserved keywords: Same definition as the one when the ANSI mode enabled.
Expand Down Expand Up @@ -88,7 +88,6 @@ Below is a list of all the keywords in Spark SQL.
<tr><td>DATABASE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DATABASES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DAY</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>DAYS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to https://www.postgresql.org/docs/current/sql-keywords-appendix.html , unit-S is not keyword.

<tr><td>DBPROPERTIES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DEFINED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DELETE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
Expand Down Expand Up @@ -136,7 +135,6 @@ Below is a list of all the keywords in Spark SQL.
<tr><td>GROUPING</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HAVING</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOUR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOURS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>IF</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>IGNORE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>IMPORT</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Expand Down Expand Up @@ -174,15 +172,9 @@ Below is a list of all the keywords in Spark SQL.
<tr><td>MAP</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MATCHED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MERGE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MICROSECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MICROSECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MILLISECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MILLISECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to https://www.postgresql.org/docs/current/sql-keywords-appendix.html , they are not keyword.

<tr><td>MINUS</td><td>reserved</td><td>strict-non-reserved</td><td>non-reserved</td></tr>
<tr><td>MINUTE</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MINUTES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MONTH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MONTHS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MSCK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>NAMESPACE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>NAMESPACES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Expand Down Expand Up @@ -242,7 +234,6 @@ Below is a list of all the keywords in Spark SQL.
<tr><td>ROWS</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SCHEMA</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>SECOND</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>SELECT</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SEMI</td><td>reserved</td><td>strict-non-reserved</td><td>non-reserved</td></tr>
<tr><td>SEPARATED</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Expand Down Expand Up @@ -293,12 +284,9 @@ Below is a list of all the keywords in Spark SQL.
<tr><td>USING</td><td>reserved</td><td>strict-non-reserved</td><td>reserved</td></tr>
<tr><td>VALUES</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>VIEW</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>WEEK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to https://www.postgresql.org/docs/current/sql-keywords-appendix.html , this is not keyword.

<tr><td>WEEKS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>WHEN</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WHERE</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WINDOW</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WITH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>YEAR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>YEARS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
</table>
52 changes: 2 additions & 50 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -813,7 +813,6 @@ booleanValue

interval
: INTERVAL (errorCapturingMultiUnitsInterval | errorCapturingUnitToUnitInterval)?
| {SQL_standard_keyword_behavior}? (errorCapturingMultiUnitsInterval | errorCapturingUnitToUnitInterval)
;

errorCapturingMultiUnitsInterval
Expand All @@ -839,23 +838,12 @@ intervalValue

intervalUnit
: DAY
| DAYS
| HOUR
| HOURS
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTE
| MINUTES
| MONTH
| MONTHS
| SECOND
| SECONDS
| WEEK
| WEEKS
| YEAR
| YEARS
| identifier
;

colPosition
Expand Down Expand Up @@ -991,7 +979,7 @@ number
// function, alias, etc.
// - Non-reserved keywords:
// Keywords that have a special meaning only in particular contexts and can be used as
// identifiers in other contexts. For example, `SELECT 1 WEEK` is an interval literal, but WEEK
// identifiers in other contexts. For example, `EXPLAIN SELECT ...` is a command, but EXPLAIN
// can be used as identifiers in other places.
// You can find the full keywords list by searching "Start of the keywords list" in this file.
// The non-reserved keywords are listed below. Keywords not in this list are reserved keywords.
Expand Down Expand Up @@ -1029,7 +1017,6 @@ ansiNonReserved
| DATA
| DATABASE
| DATABASES
| DAYS
| DBPROPERTIES
| DEFINED
| DELETE
Expand Down Expand Up @@ -1060,7 +1047,6 @@ ansiNonReserved
| FUNCTIONS
| GLOBAL
| GROUPING
| HOURS
| IF
| IGNORE
| IMPORT
Expand Down Expand Up @@ -1089,12 +1075,6 @@ ansiNonReserved
| MAP
| MATCHED
| MERGE
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTES
| MONTHS
| MSCK
| NAMESPACE
| NAMESPACES
Expand Down Expand Up @@ -1141,7 +1121,6 @@ ansiNonReserved
| ROW
| ROWS
| SCHEMA
| SECONDS
| SEPARATED
| SERDE
| SERDEPROPERTIES
Expand Down Expand Up @@ -1179,10 +1158,7 @@ ansiNonReserved
| USE
| VALUES
| VIEW
| WEEK
| WEEKS
| WINDOW
| YEARS
;

// When `SQL_standard_keyword_behavior=false`, there are 2 kinds of keywords in Spark SQL.
Expand Down Expand Up @@ -1264,7 +1240,6 @@ nonReserved
| DATABASE
| DATABASES
| DAY
| DAYS
| DBPROPERTIES
| DEFINED
| DELETE
Expand Down Expand Up @@ -1310,7 +1285,6 @@ nonReserved
| GROUPING
| HAVING
| HOUR
| HOURS
| IF
| IGNORE
| IMPORT
Expand Down Expand Up @@ -1344,14 +1318,8 @@ nonReserved
| MAP
| MATCHED
| MERGE
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTE
| MINUTES
| MONTH
| MONTHS
| MSCK
| NAMESPACE
| NAMESPACES
Expand Down Expand Up @@ -1408,7 +1376,6 @@ nonReserved
| ROWS
| SCHEMA
| SECOND
| SECONDS
| SELECT
| SEPARATED
| SERDE
Expand Down Expand Up @@ -1457,14 +1424,11 @@ nonReserved
| USER
| VALUES
| VIEW
| WEEK
| WEEKS
| WHEN
| WHERE
| WINDOW
| WITH
| YEAR
| YEARS
;

// NOTE: If you add a new token in the list below, you should update the list of keywords
Expand Down Expand Up @@ -1527,7 +1491,6 @@ DATA: 'DATA';
DATABASE: 'DATABASE';
DATABASES: 'DATABASES' | 'SCHEMAS';
DAY: 'DAY';
DAYS: 'DAYS';
DBPROPERTIES: 'DBPROPERTIES';
DEFINED: 'DEFINED';
DELETE: 'DELETE';
Expand Down Expand Up @@ -1574,7 +1537,6 @@ GROUP: 'GROUP';
GROUPING: 'GROUPING';
HAVING: 'HAVING';
HOUR: 'HOUR';
HOURS: 'HOURS';
IF: 'IF';
IGNORE: 'IGNORE';
IMPORT: 'IMPORT';
Expand Down Expand Up @@ -1612,14 +1574,8 @@ MACRO: 'MACRO';
MAP: 'MAP';
MATCHED: 'MATCHED';
MERGE: 'MERGE';
MICROSECOND: 'MICROSECOND';
MICROSECONDS: 'MICROSECONDS';
MILLISECOND: 'MILLISECOND';
MILLISECONDS: 'MILLISECONDS';
MINUTE: 'MINUTE';
MINUTES: 'MINUTES';
MONTH: 'MONTH';
MONTHS: 'MONTHS';
MSCK: 'MSCK';
NAMESPACE: 'NAMESPACE';
NAMESPACES: 'NAMESPACES';
Expand Down Expand Up @@ -1679,7 +1635,6 @@ ROW: 'ROW';
ROWS: 'ROWS';
SCHEMA: 'SCHEMA';
SECOND: 'SECOND';
SECONDS: 'SECONDS';
SELECT: 'SELECT';
SEMI: 'SEMI';
SEPARATED: 'SEPARATED';
Expand Down Expand Up @@ -1732,14 +1687,11 @@ USER: 'USER';
USING: 'USING';
VALUES: 'VALUES';
VIEW: 'VIEW';
WEEK: 'WEEK';
WEEKS: 'WEEKS';
WHEN: 'WHEN';
WHERE: 'WHERE';
WINDOW: 'WINDOW';
WITH: 'WITH';
YEAR: 'YEAR';
YEARS: 'YEARS';
//============================
// End of the keywords list
//============================
Expand Down
36 changes: 1 addition & 35 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -645,11 +645,6 @@ class ExpressionParserSuite extends AnalysisTest {
"-" -> UnaryMinus(expected)
).foreach { case (sign, expectedLiteral) =>
assertEqual(s"${sign}interval $intervalValue", expectedLiteral)

// SPARK-23264 Support interval values without INTERVAL clauses if ANSI SQL enabled
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
assertEqual(intervalValue, expected)
}
}
}

Expand Down Expand Up @@ -687,8 +682,7 @@ class ExpressionParserSuite extends AnalysisTest {
Literal(IntervalUtils.stringToInterval("1 second 1 millisecond")))

// Non Existing unit
intercept("interval 10 nanoseconds",
"no viable alternative at input '10 nanoseconds'")
intercept("interval 10 nanoseconds", "invalid unit 'nanoseconds'")

// Year-Month intervals.
val yearMonthValues = Seq("123-10", "496-0", "-2-3", "-123-0")
Expand Down Expand Up @@ -732,34 +726,6 @@ class ExpressionParserSuite extends AnalysisTest {
Literal(new CalendarInterval(3, 4, 22001000L)))
}

test("SPARK-23264 Interval Compatibility tests") {
def checkIntervals(intervalValue: String, expected: Literal): Unit = {
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
assertEqual(intervalValue, expected)
}

// Compatibility tests: If ANSI SQL disabled, `intervalValue` should be parsed as an alias
withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
val aliases = defaultParser.parseExpression(intervalValue).collect {
case a @ Alias(_: Literal, name)
if intervalUnits.exists { unit => name.startsWith(unit.toString) } => a
}
assert(aliases.size === 1)
}
}
val forms = Seq("", "s")
val values = Seq("5", "1", "-11", "8")
intervalUnits.foreach { unit =>
forms.foreach { form =>
values.foreach { value =>
val expected = intervalLiteral(unit, value)
checkIntervals(s"$value $unit$form", expected)
checkIntervals(s"'$value' $unit$form", expected)
}
}
}
}

test("composed expressions") {
assertEqual("1 + r.r As q", (Literal(1) + UnresolvedAttribute("r.r")).as("q"))
assertEqual("1 - f('o', o(bar))", Literal(1) - 'f.function("o", 'o.function('bar)))
Expand Down
Loading