[TASK][EASY] Spark Engine should throw an exception when the engine fails to start.

[ASiegeLion]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

In the main method of SparkSQLEngine, exceptions are caught but not thrown. If the engine fails to start, the k8s status is displayed as finished, and the kyuubi server will not exit until the engine initialization times out.

Affects Version(s)

master/1.7.1

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[pan3793]

Thanks for reporting this issue, but seems it's not limited to K8s but a generic issue, right?

PS: Please use text as much as possible, the picture is not searchable which makes the future explorer hard to find it.

[ASiegeLion]

Thanks for reporting this issue, but seems it's not limited to K8s but a generic issue, right?

PS: Please use text as much as possible, the picture is not searchable which makes the future explorer hard to find it.

If the createSpark method fails, YARN seems to be able to know, but K8S cannot. If the startEngine method fails, they don't know.

[cxzl25]

YARN seems to be able to know

If the engine does not throw an exception, the status seen in YARN is also SUCCESS.

org.apache.spark.deploy.yarn.ApplicationMaster

            mainMethod.invoke(null, userArgs.toArray)
            finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)

https://github.com/apache/spark/blob/9a44dc4f9ee005f8e82f15bd731c9c870cc4a606/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L738-L739

For example, setting a non-existent catalog leads to the initialization failure of the spark yarn cluster mode.

spark.sql.catalog.spark_catalog=NOT_EXIST_CATALOG

[ASiegeLion]

YARN seems to be able to know

If the engine does not throw an exception, the status seen in YARN is also SUCCESS.

org.apache.spark.deploy.yarn.ApplicationMaster

            mainMethod.invoke(null, userArgs.toArray)
            finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)

https://github.com/apache/spark/blob/9a44dc4f9ee005f8e82f15bd731c9c870cc4a606/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L738-L739

For example, setting a non-existent catalog leads to the initialization failure of the spark yarn cluster mode.

spark.sql.catalog.spark_catalog=NOT_EXIST_CATALOG

You are right, the exception I see is the failure to start the ApplicationMaster, not the spark Engine. So I mistakenly thought that YARN seems to be able to know