[TASK][MEDIUM] Support running JDBC engine on YARN AM

[pan3793]

What's the level of this task?

MEDIUM

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

• Knowledge of Scala, Kyuubi architecture, Apache YARN
• Kerberos and Hadoop Security (optional)

Background and Goals

Kyuubi supports running the Hive engine on YARN AM (Application Master) since 1.9.0, see #5868 and #6199, it removes the requirement of running engine process on Kyuubi Server's machine but moves to YARN cluster, just like Spark on YARN cluster mode and Flink YARN application mode.

We'd like to see similar functionalities for the JDBC engine.

Implementation steps

Refer to the Hive engine implementation to complete the code, test, CI, etc.

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program

This may help for testing https://github.com/awesome-kyuubi/hadoop-testing

[lsm1]

i want to have a try, plz assign to me if possible.

[pan3793]

@lsm1 thanks