[SPARK-33463][SQL] Keep Job Id during incremental collect in Spark Thrift Server #30390
Conversation
|
Can one of the admins verify this patch? |
| @@ -126,6 +126,17 @@ private[hive] class SparkExecuteStatementOperation( | |||
| } | |||
|
|
|||
| def getNextRowSet(order: FetchOrientation, maxRowsL: Long): RowSet = withLocalProperties { | |||
| try { | |||
| sqlContext.sparkContext.setJobGroup(statementId, statement) | |||
There was a problem hiding this comment.
minor: in execute substitutorStatement is used as a description of the jobgroup. Maybe pull out substitutorStatement to be a class field and use it here as well?
There was a problem hiding this comment.
You are right!!! Thanks a lot for taking the time of revisioning my code.
I implemented the change, hopefully now it looks better. Looking forward to hearing your thoughts.
There was a problem hiding this comment.
+1, LGTM. Thank you, @gumartinm and @juliuszsompolski .
Merged to master for Apache Spark 3.1.
dongjoon-hyun
changed the title
|
@gumartinm . I added you to the Apache Spark contributor group and assigned SPARK-33463 to you. |
What changes were proposed in this pull request?
When enabling spark.sql.thriftServer.incrementalCollect Job Ids get lost and tracing queries in Spark Thrift Server ends up being too complicated.
Why are the changes needed?
Because it will make easier tracing Spark Thrift Server queries.
Does this PR introduce any user-facing change?
No
How was this patch tested?
The current tests are enough. No need of more tests.