[SPARK-35086][SQL][CORE] --verbose should be passed to Spark SQL CLI too #32163

AngersZhuuuu · 2021-04-14T08:00:33Z

What changes were proposed in this pull request?

In current code, if we run spark sql with

./bin/spark-sql --verbose

It won't be passed to end SparkSQLCliDriver, then the SessionState won't call setIsVerbose

In the CLI option, it shows

CLI options:
 -v,--verbose                     Verbose mode (echo executed SQL to the
                                  console)

It's not consistent. This pr fix this issue

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

when user call -v when run spark sql, sql will be echoed to console.

How was this patch tested?

Added UT

AngersZhuuuu · 2021-04-14T09:01:05Z

gentle ping @yaooqinn @cloud-fan

SparkQA · 2021-04-14T09:40:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41916/

SparkQA · 2021-04-14T09:40:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41916/

HyukjinKwon · 2021-04-14T11:23:07Z

cc @wangyum too FYI

SparkQA · 2021-04-14T11:30:55Z

Test build #137337 has finished for PR 32163 at commit e7ccfa0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-04-14T13:53:51Z

Test build #748540421 for PR 32163 at commit 7ab7599.

github-actions · 2021-04-14T13:58:05Z

Test build #748540421 for PR 32163 at commit e7ccfa0.

SparkQA · 2021-04-14T14:38:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41929/

SparkQA · 2021-04-14T14:38:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41929/

yaooqinn · 2021-04-14T15:12:52Z

sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala

  }
+
+  test("SPARK-35086: --verbose should be passed to Spark SQL CLI") {
+    runCliWithin(2.minute,


nit: inline, runCliWithin(2.minute, Seq("--verbose"))("SELECT 1;" -> "SELECT 1")

SparkQA · 2021-04-14T16:44:19Z

Test build #137350 has finished for PR 32163 at commit 195c933.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-04-14T17:25:39Z

Test build #748540421 for PR 32163 at commit 195c933.

github-actions · 2021-04-15T01:10:10Z

Test build #750310558 for PR 32163 at commit 4d88408.

SparkQA · 2021-04-15T02:37:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41961/

SparkQA · 2021-04-15T02:37:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41961/

SparkQA · 2021-04-15T04:24:12Z

Test build #137384 has finished for PR 32163 at commit 4d88408.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2021-04-15T05:00:19Z

Merged to master.

HyukjinKwon · 2021-04-18T03:09:39Z

The JIRA number here was incorrect. It had to be SPARK-35068.

HyukjinKwon · 2021-04-18T03:12:47Z

I switched the JIRA between SPARK-35068 and SPARK-35086. cc @MaxGekk FYI

LuciferYang · 2021-10-19T02:57:26Z

We build a client with v3.2.0 and run bin/spark-shell --master local --verbose, the spark-shell startup failed now @AngersZhuuuu @cloud-fan @yaooqinn @dongjoon-hyun @HyukjinKwon @wangyum ,Is this expected?

Also ping @toujours33

Main class:
org.apache.spark.repl.Main
Arguments:
--verbose
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,local)
Classpath elements:



Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
bad option: '--verbose'

LuciferYang · 2021-10-19T02:59:25Z

And bin/spark-shell --master local --verbose can startup spark-shell succesfuls use Spark 3.1.2

AngersZhuuuu · 2021-10-19T03:07:36Z

We build a client with v3.2.0 and run bin/spark-shell --master local --verbose, the spark-shell startup failed now @AngersZhuuuu @cloud-fan @yaooqinn @dongjoon-hyun @HyukjinKwon @wangyum ,Is this expected?

Also ping @toujours33
Main class:
org.apache.spark.repl.Main
Arguments:
--verbose
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,local)
Classpath elements:



Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
bad option: '--verbose'

Checking this.

LuciferYang · 2021-10-19T03:10:44Z

Thanks @AngersZhuuuu

HyukjinKwon · 2021-10-19T03:15:50Z

Thanks @LuciferYang and @AngersZhuuuu!

yaooqinn · 2021-10-19T03:16:19Z

Looks like a bug, thanks @LuciferYang and @AngersZhuuuu!

AngersZhuuuu · 2021-10-19T03:27:43Z

Found the reason, raise a fix pr soon and with a spark-shell verbose unit test

… class when is sql shell ### What changes were proposed in this pull request? In #32163 spark pass `--verbose` to main class o support spark-sql shell can use verbose argument too. But for other shell main class such as saprk-shell, it's intercepter don't support `--verbose`, so we should only pass `--verbose` for sql shell ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Closes #34322 from AngersZhuuuu/SPARK-37052. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

… class when is sql shell ### What changes were proposed in this pull request? In #32163 spark pass `--verbose` to main class o support spark-sql shell can use verbose argument too. But for other shell main class such as saprk-shell, it's intercepter don't support `--verbose`, so we should only pass `--verbose` for sql shell ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Closes #34322 from AngersZhuuuu/SPARK-37052. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a6d3a2c) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

… class when is sql shell ### What changes were proposed in this pull request? In apache#32163 spark pass `--verbose` to main class o support spark-sql shell can use verbose argument too. But for other shell main class such as saprk-shell, it's intercepter don't support `--verbose`, so we should only pass `--verbose` for sql shell ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Closes apache#34322 from AngersZhuuuu/SPARK-37052. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a6d3a2c) Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 34086b0) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

… class when is sql shell ### What changes were proposed in this pull request? In apache#32163 spark pass `--verbose` to main class o support spark-sql shell can use verbose argument too. But for other shell main class such as saprk-shell, it's intercepter don't support `--verbose`, so we should only pass `--verbose` for sql shell ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Closes apache#34322 from AngersZhuuuu/SPARK-37052. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a6d3a2c) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

AngersZhuuuu added 3 commits April 14, 2021 15:45

[SPARK-35086][SQL][CORE] --verbose should be passed to Spark SQL CLI too

547e92e

Update CliSuite.scala

7ab7599

Update CliSuite.scala

e7ccfa0

Update CliSuite.scala

195c933

github-actions bot added CORE SQL labels Apr 14, 2021

cloud-fan approved these changes Apr 14, 2021

View reviewed changes

dongjoon-hyun approved these changes Apr 14, 2021

View reviewed changes

yaooqinn reviewed Apr 14, 2021

View reviewed changes

yaooqinn approved these changes Apr 14, 2021

View reviewed changes

Update CliSuite.scala

4d88408

wangyum approved these changes Apr 15, 2021

View reviewed changes

wangyum closed this in 9b2e0d6 Apr 15, 2021

AngersZhuuuu mentioned this pull request Oct 19, 2021

[SPARK-37052][CORE] Spark should only pass --verbose argument to main class when is sql shell #34322

Closed

[SPARK-35086][SQL][CORE] --verbose should be passed to Spark SQL CLI too #32163

[SPARK-35086][SQL][CORE] --verbose should be passed to Spark SQL CLI too #32163

Uh oh!

Conversation

AngersZhuuuu commented Apr 14, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

HyukjinKwon commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

github-actions bot commented Apr 14, 2021

Uh oh!

github-actions bot commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

yaooqinn Apr 14, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

github-actions bot commented Apr 14, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

wangyum commented Apr 15, 2021

Uh oh!

HyukjinKwon commented Apr 18, 2021

Uh oh!

HyukjinKwon commented Apr 18, 2021

Uh oh!

LuciferYang commented Oct 19, 2021

Uh oh!

LuciferYang commented Oct 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AngersZhuuuu commented Oct 19, 2021

Uh oh!

LuciferYang commented Oct 19, 2021

Uh oh!

HyukjinKwon commented Oct 19, 2021

Uh oh!

yaooqinn commented Oct 19, 2021

Uh oh!

AngersZhuuuu commented Oct 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

LuciferYang commented Oct 19, 2021 •

edited

Loading