[Umbrella] Spark Configuration Validation

[yaooqinn]

Describe the proposal

When a SparkConf is invalid for launching our engines, it is better to fail fast instead of waiting for Spark to check while engine bootstrapping.

Motivation

1. Save time to create an invalid engine
2. Avoid possible hang
3. better error message

Here is an example I did for Spark to validate its own configs when talking to Kubernetes. apache/spark#32610

Task list

Generally speaking, we can have 1) a tool in kyuubi-ctl to check server/engine conf in configuration files are valid, 2) The logic of the tool can also be called when engine bootstrapping for the configurations from the client side

• Check if spark.kubernetes.executor.podNamePrefix is invalid #833
• Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value #1018

[ulysses-you]

FYI some working in Apache Spark apache/spark#33491