[SPARK-8552] [THRIFTSERVER] Using incorrect database in multiple sessions #7118

navis · 2015-06-30T08:05:44Z

Currently, metaHive in HiveContext shares single SessionState instance with all execution threads, which makes problems especially with "use" command. I've also went into this problem and spent hard time to know what has happened. This is just first try and need more tests.

marmbrus · 2015-07-06T22:04:29Z

ok to test

SparkQA · 2015-07-06T22:48:47Z

Test build #36606 has finished for PR 7118 at commit f5a6830.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)

SparkQA · 2015-08-21T00:48:05Z

Test build #41336 has finished for PR 7118 at commit 5f3240a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-21T01:05:40Z

Test build #41339 has finished for PR 7118 at commit f1d01d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-21T08:03:27Z

Test build #41358 has finished for PR 7118 at commit cd9aaf0.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-21T09:48:38Z

Test build #41363 has finished for PR 7118 at commit c707370.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-24T02:11:54Z

Test build #41434 has finished for PR 7118 at commit 21b42cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-24T03:38:43Z

Test build #41436 has finished for PR 7118 at commit 8691a2c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-24T10:59:25Z

Test build #41449 has finished for PR 7118 at commit dac944b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-25T04:02:24Z

Test build #41506 has finished for PR 7118 at commit f64398b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-26T03:24:51Z

Test build #41578 has finished for PR 7118 at commit b3f7805.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-08-31T11:53:56Z

Test build #41819 timed out for PR 7118 at commit eaeb692 after a configured wait of 250m.

SparkQA · 2015-09-08T08:10:02Z

Test build #42124 has finished for PR 7118 at commit a7baede.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-09T01:57:04Z

Test build #42167 has finished for PR 7118 at commit 28c7649.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-09-09T03:23:04Z

Test build #42175 has finished for PR 7118 at commit 6a16a7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

WangTaoTheTonic · 2015-09-10T07:16:41Z

I tested this patch, but got this error when executing "show databases;" using beeline,

15/09/10 15:11:02 INFO SessionState: Created HDFS directory: /tmp/hive/root/fc5c8bbe-0e63-49f1-8286-ec51c4432b94/_tmp_space.db
15/09/10 15:11:02 INFO HiveSessionImpl: Operation log session directory is created: /tmp/root/operation_logs/fc5c8bbe-0e63-49f1-8286-ec51c4432b94
15/09/10 15:11:26 ERROR SparkExecuteStatementOperation: Error running hive query as user : root
java.lang.NullPointerException
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:182)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

navis · 2015-09-10T09:11:57Z

@WangTaoTheTonic I've also seen that rebasing this to spark-1.5.0. Seemed done something wrong in rebase process. I'll update soon.

SparkQA · 2015-09-11T06:02:06Z

Test build #42312 has finished for PR 7118 at commit 7eba459.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
- class CoGroupedRDD[K: ClassTag](
- class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](
- class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, HasHandleInvalid):
- class HasHandleInvalid(Params):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

WangTaoTheTonic · 2015-09-11T10:05:00Z

@navis Thanks you for the fix. I have tested "use $database" on local Thrift Server and the function is ok.

There might still have a test failed need to fix.

WangTaoTheTonic · 2015-09-12T03:28:03Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

The comments need to be updated with its construct args.

…ions

navis · 2015-09-14T01:08:53Z

@WangTaoTheTonic I've fixed test itself, which seemed invalid. "add jar" should be applied to the session which called it but in test, it expects added jar to be accessed with other sessions.

SparkQA · 2015-09-14T02:44:30Z

Test build #42388 has finished for PR 7118 at commit 03d4370.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-09-14T02:53:45Z

Test build #42387 has finished for PR 7118 at commit a72d490.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

SparkQA · 2015-09-14T11:07:09Z

Test build #42418 has finished for PR 7118 at commit 431b0d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SQLContext(
- class HiveContext(sc: SparkContext, optionConf: Map[String, String] = Map.empty)
- protected[hive] class SQLSession(val sessionID: Int, val config: Map[String, String])
- class SharedWrapper(
- class IsolatedWrapper(

WangTaoTheTonic · 2015-09-29T11:45:39Z

@marmbrus How do you think this fix? As the issue priority is very high, I think we better fix it ASAP.

marmbrus · 2015-09-29T17:35:57Z

I think I prefer the solution taken in #8909, which limits the use of thread local variables

davies · 2015-10-09T00:48:36Z

Since #8909 is merged, would you mind close this PR?

pzzs · 2015-11-07T08:18:37Z

@navis
when i use spark-sql and run sql like that
"
add jar /home/udf-0.0.1-SNAPSHOT.jar;
create temporary function arr_greater_equal as 'com.xx.yy.dac.hive.udf.UDFArrayGreaterEqual';
"
I got a error
15/11/07 16:01:15 ERROR CliDriver: org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException [Error 10072]: Database does not exist: default
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:349)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:326)
at org.apache.spark.sql.hive.client.SharedWrapper.withHiveState(ClientWrapper.scala:469)
at org.apache.spark.sql.hive.HiveContext$$anon$2.withHiveState(HiveContext.scala:194)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:138)
at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:326)
at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:316)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:542)
at org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)

navis force-pushed the SPARK-8552 branch from d13b8e2 to f5a6830 Compare June 30, 2015 08:44

navis force-pushed the SPARK-8552 branch from f5a6830 to 5f3240a Compare August 21, 2015 00:42

navis force-pushed the SPARK-8552 branch from 5f3240a to f1d01d5 Compare August 21, 2015 00:53

navis force-pushed the SPARK-8552 branch from c707370 to 21b42cc Compare August 24, 2015 01:41

navis force-pushed the SPARK-8552 branch from f64398b to b3f7805 Compare August 26, 2015 01:01

navis force-pushed the SPARK-8552 branch from b3f7805 to eaeb692 Compare August 31, 2015 07:33

navis force-pushed the SPARK-8552 branch from eaeb692 to a7baede Compare September 8, 2015 07:38

navis force-pushed the SPARK-8552 branch 2 times, most recently from 28c7649 to 6a16a7e Compare September 9, 2015 01:06

navis force-pushed the SPARK-8552 branch from 6a16a7e to 7eba459 Compare September 11, 2015 05:09

WangTaoTheTonic reviewed Sep 12, 2015
View reviewed changes

navis added 10 commits September 14, 2015 09:58

[SPARK-8552] [THRIFTSERVER] Using incorrect database in multiple sess…

da60597

…ions

Fixed scalastyle failure

ea51c91

Fixed MIMA failure

73fff75

defaultSession to lazy val

1f38990

Fixed SQLConfSuite failure

893a7c8

Mistakenly overrided cancel opration

8028f6a

Added common.logging to shared class list

f06f41a

fixed test fails

b0716fe

Fixed NPE and other problems

7fad802

yet more fix

a72d490

navis force-pushed the SPARK-8552 branch from 7eba459 to a72d490 Compare September 14, 2015 00:59

fixed javadoc

03d4370

fix test fails

431b0d1

navis mentioned this pull request Oct 7, 2015

[SPARK-10810] [SPARK-10902] [SQL] Improve session management in SQL #8909

Closed

navis closed this Oct 12, 2015

[SPARK-8552] [THRIFTSERVER] Using incorrect database in multiple sessions #7118

[SPARK-8552] [THRIFTSERVER] Using incorrect database in multiple sessions #7118

Uh oh!

Conversation

navis commented Jun 30, 2015

Uh oh!

marmbrus commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

SparkQA commented Aug 21, 2015

Uh oh!

SparkQA commented Aug 21, 2015

Uh oh!

SparkQA commented Aug 21, 2015

Uh oh!

SparkQA commented Aug 21, 2015

Uh oh!

SparkQA commented Aug 24, 2015

Uh oh!

SparkQA commented Aug 24, 2015

Uh oh!

SparkQA commented Aug 24, 2015

Uh oh!

SparkQA commented Aug 25, 2015

Uh oh!

SparkQA commented Aug 26, 2015

Uh oh!

SparkQA commented Aug 31, 2015

Uh oh!

SparkQA commented Sep 8, 2015

Uh oh!

SparkQA commented Sep 9, 2015

Uh oh!

SparkQA commented Sep 9, 2015

Uh oh!

WangTaoTheTonic commented Sep 10, 2015

Uh oh!

navis commented Sep 10, 2015

Uh oh!

SparkQA commented Sep 11, 2015

Uh oh!

WangTaoTheTonic commented Sep 11, 2015

Uh oh!

WangTaoTheTonic Sep 12, 2015

Choose a reason for hiding this comment

Uh oh!

navis commented Sep 14, 2015

Uh oh!

SparkQA commented Sep 14, 2015

Uh oh!

SparkQA commented Sep 14, 2015

Uh oh!

SparkQA commented Sep 14, 2015

Uh oh!

WangTaoTheTonic commented Sep 29, 2015

Uh oh!

marmbrus commented Sep 29, 2015

Uh oh!

davies commented Oct 9, 2015

Uh oh!

pzzs commented Nov 7, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants