[SPARK-31892][SQL][FOLLOWUP][test-java11] Improve test coverage for datetime pattern with formatter functions #28718

yaooqinn · 2020-06-03T09:21:13Z

What changes were proposed in this pull request?

This PR is a followup for #28706, we want fully check for the formatter, which can warn us ahead if there is any behavior change when we update JDK.

Why are the changes needed?

test coverage and hotfix for the silly mistake in #28706 at https://github.com/apache/spark/pull/28718/files#diff-7f589e01d3e5e5ea284c1622527d4984R49

Does this PR introduce any user-facing change?

NO

How was this patch tested?

add tests

…formatting

yaooqinn · 2020-06-03T10:30:57Z

sql/core/src/test/resources/sql-tests/results/datetime.sql.out

+-- !query schema
+struct<date_format(t, VV z zz zzz zzzz O OOOO X XX XXX XXXX XXXXX x xx xxx xxxx xxxx xxxxx Z ZZ ZZZ ZZZZ ZZZZZ):string>
+-- !query output
+America/Los_Angeles PST PST PST Pacific Standard Time GMT-7:52:58 GMT-07:52:58 -0752 -0752 -07:52 -075258 -07:52:58 -0752 -0752 -07:52 -075258 -075258 -07:52:58 -0752 -0752 -0752 GMT-07:52:58 -07:52:58


Do you know what causes this inaccuracy for dates before 1582-10-15？@cloud-fan

cc @MaxGekk

It seems to only display the value of spark.sql.session.timeZone

actually, this is expected. The Spark test uses timezone America/Los_Angeles, which is indeed GMT-7:52:58 long time before, see https://www.politico.com/story/2017/11/18/railroads-initiative-synchronizes-us-clocks-nov-18-1883-244935

Spark's timestamp value doesn't have an associated timezone, so extracting timezone means showing the session local timezone.

...atalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala

sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out

sql/core/src/test/resources/sql-tests/inputs/datetime.sql

...atalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting.sql

SparkQA · 2020-06-03T14:34:33Z

Test build #123479 has finished for PR 28718 at commit 6449e08.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T15:07:17Z

Test build #123476 has finished for PR 28718 at commit 7e59d65.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T15:11:45Z

Test build #123487 has finished for PR 28718 at commit 47d7322.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T15:15:52Z

Test build #123480 has finished for PR 28718 at commit de63621.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T15:30:10Z

Test build #123482 has finished for PR 28718 at commit 3fea4de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T16:55:45Z

Test build #123490 has finished for PR 28718 at commit 83499a1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql

SparkQA · 2020-06-03T17:03:24Z

Test build #123486 has finished for PR 28718 at commit 34aef79.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

cloud-fan · 2020-06-03T17:30:44Z

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql

@@ -0,0 +1,36 @@
+--- TESTS FOR DATETIME FORMATTING FUNCTIONS WITH INVALID PATTERNS ---
+
+-- separating this from datetime-formatting.sql ,because the text form


.sql ,because -> .sql, because

cloud-fan · 2020-06-03T17:31:27Z

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql

+select date_format('2018-11-17 13:33:33.333', 'EEEEE');
+select date_format('2018-11-17 13:33:33.333', 'FF');
+select date_format('2018-11-17 13:33:33.333', 'ddd');
+-- DD is invalid in 8, but valid in 11 for the JDKs that PR builder uses


DD is invalid if the day-of-year exceeds 100, but it becomes valid in Java 11.

cloud-fan · 2020-06-03T17:32:17Z

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting.sql

+
+select col, date_format(col, 'S SS SSS SSSS SSSSS SSSSSS SSSSSSS SSSSSSSS SSSSSSSSS') from v;
+
+-- add upper function here to avoid


to avoid what?

cloud-fan · 2020-06-03T18:15:43Z

@dongjoon-hyun do you know how to trigger java 11 test in the PR builder?

MaxGekk · 2020-06-03T18:54:37Z

@cloud-fan See the example #28463 - add [test-java11]

cloud-fan · 2020-06-03T19:18:43Z

retest this please

dongjoon-hyun · 2020-06-03T19:29:50Z

Sorry for being late. Yes, @MaxGekk 's comment is correct.

SparkQA · 2020-06-03T19:44:16Z

Test build #123488 has finished for PR 28718 at commit 1e78a4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T21:30:17Z

Test build #123491 has finished for PR 28718 at commit 4d34f44.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T21:55:09Z

Test build #123498 has finished for PR 28718 at commit eeceba5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T23:15:32Z

Test build #123494 has finished for PR 28718 at commit eeceba5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T23:25:23Z

Test build #123493 has finished for PR 28718 at commit 11c9653.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

[SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …

7e59d65

…formatting

probot-autolabeler bot added the SQL label Jun 3, 2020

yaooqinn added 3 commits June 3, 2020 17:32

add case

6449e08

remove default value

de63621

better look

3fea4de

yaooqinn commented Jun 3, 2020

View reviewed changes

cloud-fan reviewed Jun 3, 2020

View reviewed changes

...atalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jun 3, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out Outdated Show resolved Hide resolved

cloud-fan reviewed Jun 3, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/datetime.sql Outdated Show resolved Hide resolved

refine

34aef79

cloud-fan reviewed Jun 3, 2020

View reviewed changes

...atalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala Show resolved Hide resolved

cloud-fan reviewed Jun 3, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting.sql Outdated Show resolved Hide resolved

yaooqinn added 2 commits June 3, 2020 22:12

Merge branch 'master' into SPARK-31892-F

47d7322

fix tests

1e78a4e

yaooqinn added 2 commits June 4, 2020 00:03

new tests

83499a1

update test

4d34f44

cloud-fan reviewed Jun 3, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql Outdated Show resolved Hide resolved

cloud-fan reviewed Jun 3, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql Outdated Show resolved Hide resolved

address comments

11c9653

yaooqinn changed the title ~~[SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …~~ [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for datetime pattern with formatter functions Jun 3, 2020

cloud-fan reviewed Jun 3, 2020

View reviewed changes

style

eeceba5

cloud-fan approved these changes Jun 3, 2020

View reviewed changes

cloud-fan changed the title ~~[SPARK-31892][SQL][FOLLOWUP] Improve test coverage for datetime pattern with formatter functions~~ [SPARK-31892][SQL][FOLLOWUP][[test-java11]] Improve test coverage for datetime pattern with formatter functions Jun 3, 2020

cloud-fan changed the title ~~[SPARK-31892][SQL][FOLLOWUP][[test-java11]] Improve test coverage for datetime pattern with formatter functions~~ [SPARK-31892][SQL][FOLLOWUP][test-java11] Improve test coverage for datetime pattern with formatter functions Jun 3, 2020

yaooqinn closed this Jun 5, 2020

		@@ -0,0 +1,36 @@
		--- TESTS FOR DATETIME FORMATTING FUNCTIONS WITH INVALID PATTERNS ---

		-- separating this from datetime-formatting.sql ,because the text form


		select col, date_format(col, 'S SS SSS SSSS SSSSS SSSSSS SSSSSSS SSSSSSSS SSSSSSSSS') from v;

		-- add upper function here to avoid

[SPARK-31892][SQL][FOLLOWUP][test-java11] Improve test coverage for datetime pattern with formatter functions #28718

[SPARK-31892][SQL][FOLLOWUP][test-java11] Improve test coverage for datetime pattern with formatter functions #28718

Uh oh!

Conversation

yaooqinn commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yaooqinn Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

cloud-fan Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jun 3, 2020

Uh oh!

MaxGekk commented Jun 3, 2020

Uh oh!

cloud-fan commented Jun 3, 2020

Uh oh!

dongjoon-hyun commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

SparkQA commented Jun 3, 2020

Uh oh!

Uh oh!

yaooqinn commented Jun 3, 2020 •

edited

Loading