Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value

[yaooqinn]

1. Describe the feature

In apache/spark#33904, I faced a performance issue with the topK scenario where K is very large which can be avoided by setting spark.sql.execution.topKSortFallbackThreshold less than the K here.

For Kyuubi users who run SQL only, improper topK SQLs will hold engines for quite a long time and they don't know why.

With a proper value for spark.sql.execution.topKSortFallbackThreshold is more suitable for Kyuubi's cases

I'd suggest this value be set to 10000

2. Motivation

3. Describe the solution

4. Additional context

[byyue]

Hi, could you please elaborate a little bit more on how to implement this?

[yaooqinn]

diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index 6968e07c..9c2832c5 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -96,6 +96,7 @@ object SparkSQLEngine extends Logging {

   def createSpark(): SparkSession = {
     val sparkConf = new SparkConf()
+    sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", "10000")
     sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
     sparkConf.setIfMissing("spark.master", "local")
     sparkConf.setIfMissing("spark.ui.port", "0")

[byyue]

Thanks! I think spark.sql.execution.topKSortFallbackThreshold is now a property name of Spark Runtime SQL Configuration?