[SPARK-34977][SQL] LIST FILES/JARS/ARCHIVES cannot handle multiple arguments properly when at least one path is quoted #32074

sarutak · 2021-04-07T05:40:31Z

What changes were proposed in this pull request?

This PR fixes an issue that LIST FILES/JARS/ARCHIVES path1 path2 ... cannot list all paths if at least one path is quoted.
An example here.

ADD FILE /tmp/test1;
ADD FILE /tmp/test2;

LIST FILES /tmp/test1 /tmp/test2;
file:/tmp/test1
file:/tmp/test2

LIST FILES /tmp/test1 "/tmp/test2";
file:/tmp/test2

In this example, the second LIST FILES doesn't show file:/tmp/test1.

To resolve this issue, I modified the syntax rule to be able to handle this case.
I also changed SparkSQLParser to be able to handle paths which contains white spaces.

Why are the changes needed?

This is a bug.
I also have a plan which extends ADD FILE/JAR/ARCHIVE to take multiple paths like Hive and the syntax rule change is necessary for that.

Does this PR introduce any user-facing change?

Yes. Users can pass quoted paths when using ADD FILE/JAR/ARCHIVE.

How was this patch tested?

New test.

…ot list all paths if at least one path is quoted.

sarutak · 2021-04-07T05:43:24Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

+      TestUtils.createJar(Seq(file8), jarFile5)
+
+      sql(s"ADD JAR ${jarFile4.getAbsolutePath}")
+      sql(s"ADD JAR ${jarFile5.getAbsolutePath}")


Unlike ADD FILE "path" and ADD ARCHIVE "path", we cannot execute ADD JAR "path" when the path contains whitespaces.
I think it's a bug and #32052 will fix this issue.

Got it. @sarutak .

SparkQA · 2021-04-07T06:33:44Z

Test build #136998 has finished for PR 32074 at commit c5a6345.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-07T07:01:55Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41575/

sarutak · 2021-04-07T07:06:21Z

retest this please.

SparkQA · 2021-04-07T07:34:00Z

Test build #137011 has finished for PR 32074 at commit c5a6345.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-07T10:02:11Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41589/

SparkQA · 2021-04-07T16:56:54Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41604/

dongjoon-hyun · 2021-04-07T18:44:04Z

#32052 is merged now. Could you rebase this to the master, @sarutak ?

SparkQA · 2021-04-07T20:39:36Z

Test build #137026 has finished for PR 32074 at commit 752da21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…t-files-bug

Add test for LIST JARS path where the path contains whitespaces.

SparkQA · 2021-04-08T04:32:05Z

Test build #137053 has finished for PR 32074 at commit b4cb445.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-08T05:27:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41632/

SparkQA · 2021-04-08T05:27:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41632/

sarutak · 2021-04-09T01:08:59Z

retest this please.

SparkQA · 2021-04-09T02:38:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41688/

SparkQA · 2021-04-09T02:38:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41688/

SparkQA · 2021-04-09T05:39:44Z

Test build #137110 has finished for PR 32074 at commit b4cb445.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…t-files-bug

SparkQA · 2021-04-09T07:23:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41695/

SparkQA · 2021-04-09T07:23:54Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41695/

SparkQA · 2021-04-09T11:35:10Z

Test build #137116 has finished for PR 32074 at commit 7416724.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UpdateStarAction(condition: Option[Expression]) extends MergeAction
case class InsertStarAction(condition: Option[Expression]) extends MergeAction

sarutak · 2021-04-12T04:08:37Z

cc: @HyukjinKwon

sql/core/src/main/scala/org/apache/spark/sql/execution/command/resources.scala

SparkQA · 2021-04-12T10:20:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41787/

SparkQA · 2021-04-12T10:20:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41787/

SparkQA · 2021-04-12T14:14:20Z

Test build #137208 has finished for PR 32074 at commit 3f78c43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Looks okay

sarutak · 2021-04-14T01:35:17Z

Thanks all. Merged into master.

Fix an issue that LIST {FILES/JARS/ARCHIVES} path1, path2, ... cann…

c5a6345

…ot list all paths if at least one path is quoted.

sarutak commented Apr 7, 2021

View reviewed changes

github-actions bot added the SQL label Apr 7, 2021

Modify the syntax rule.

752da21

sarutak added 2 commits April 8, 2021 10:01

Merge branch 'master' of https://github.com/apache/spark into fix-lis…

c45c715

…t-files-bug

Keep the compatibility with SPARK-34977 for ADD FILE/JAR/ARCHIVE

b4cb445

Add test for LIST JARS path where the path contains whitespaces.

Merge branch 'master' of https://github.com/apache/spark into fix-lis…

7416724

…t-files-bug

HyukjinKwon reviewed Apr 12, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/resources.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Apr 12, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/resources.scala Show resolved Hide resolved

HyukjinKwon reviewed Apr 12, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/resources.scala Show resolved Hide resolved

Fix wrong indentation.

3f78c43

HyukjinKwon reviewed Apr 13, 2021

View reviewed changes

sarutak closed this in ef05e89 Apr 14, 2021

[SPARK-34977][SQL] LIST FILES/JARS/ARCHIVES cannot handle multiple arguments properly when at least one path is quoted #32074

[SPARK-34977][SQL] LIST FILES/JARS/ARCHIVES cannot handle multiple arguments properly when at least one path is quoted #32074

Uh oh!

Conversation

sarutak commented Apr 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak Apr 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 7, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

sarutak commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

dongjoon-hyun commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 7, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

sarutak commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

sarutak commented Apr 12, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Apr 12, 2021

Uh oh!

SparkQA commented Apr 12, 2021

Uh oh!

SparkQA commented Apr 12, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

sarutak commented Apr 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sarutak commented Apr 7, 2021 •

edited

Loading

sarutak Apr 7, 2021 •

edited

Loading