[TASK][EASY] Add a common getSessionConf method in spark engine module

[wForget]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Knowledge of Kyuubi
• Familiar with the open-source ecosystem

Background and Goals

As discussed in #5410 (comment), some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so we can implement a common method to get session-level configurations.

Implementation steps

Add a common method in org.apache.kyuubi.engine.spark.KyuubiSparkUtil, like:

  def getSessionConf[T](configEntry: ConfigEntry[T], spark: SparkSession): T = {
    spark.conf.getOption(configEntry.key).map(configEntry.valueConverter).getOrElse {
      SparkSQLEngine.kyuubiConf.get(configEntry)
    }
  }

Refactor the code that gets configurations using getSessionConf method.

Additional context

No response

[davidyuan1223]

i think i can try improvement this, then fix the #5410's test case

[wForget]

i think i can try improvement this, then fix the #5410's test case

thanks, assigned