[SPARK-36659][SQL] Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config

[yaooqinn]

What changes were proposed in this pull request?

Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config

Why are the changes needed?

spark.sql.execution.topKSortFallbackThreshold now is an internal config hidden from users Integer.MAX_VALUE - 15 as its default. In many real-world cases, if the K is very big, there would be performance issues.

It's better to leave this choice to users

Does this PR introduce any user-facing change?

spark.sql.execution.topKSortFallbackThreshold is now user-facing

How was this patch tested?

passing GA

[yaooqinn]

cc @cloud-fan

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47453/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47453/

[SparkQA]

Test build #142953 has finished for PR 33904 at commit 25cadd5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yaooqinn]

thanks, merged to master

[dongjoon-hyun]

+1, late LGTM.
This is only for Apache Spark 3.3, @yaooqinn ?

[dongjoon-hyun]

For Apache Kyuubi, I guess we may land this to branch-3.2 before Apache Spark 3.2.0 RC3 starts. In any way, this config was there for a long time already.

cc @gengliangwang

[yaooqinn]

+1, late LGTM.
This is only for Apache Spark 3.3, @yaooqinn ?

@dongjoon-hyun thanks
Oh, I merged it only to 3.3. But I am quite OK to add it into 3.2

[dongjoon-hyun]

Thank you, @yaooqinn . I backported to branch-3.2.