[Bug] Kyuubi job status always shows as "Succeeded" even when the actual job has failed

[sohurdc]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

Kyuubi-1.10.2 and spark-3.5.6.
Kyuubi job status always shows as "Succeeded" even when the actual job has failed.
My execution script is as follows:
beeline -u "jdbc:hive2://kyuubiserver.my.com:10009/myuser;principal=hive/kyuubi.my.com@MY.COM;hive.server2.proxy.user=myuser" --hiveconf spark.yarn.queue=center --hiveconf spark.executor.memory=4g -e "insert overwrite directory 'hdfs:///user/myuser/tmp/2025-07-23' select * from ..."

then job faied with error such as:
Caused by: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Permission denied: user [myuser] does not have [write] privilege on ...

But the final job status shown on the Spark log page is "Succeeded", which makes it impossible to determine the actual job outcome based on the job status. Job status monitoring and retry mechanisms become completely unusable.

Affects Version(s)

1.10.2

Kyuubi Server Log Output

kyuubi server log:
2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: RUNNING_STAT
E -> ERROR_STATE, time taken: 0.439 seconds
2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta
tement state=ERROR user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: ERROR_STA
TE -> CLOSED_STATE, time taken: 0.511 seconds
2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta
tement state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.441 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THand
leIdentifier(guid:71 C3 2A 89 BF 36 40 5B A5 81 3C A2 B2 A6 20 2E, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engin
e side
2025-08-01 10:50:48.471 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [5d12d959-c7a6-4dfc-8
c18-6a08baf63fbd]
2025-08-01 10:50:48.473 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.session.KyuubiSessionManager: myuser's KyuubiSessionImpl with SessionHandle [5d12d959-c7a6-4dfc-8c18-6
a08baf63fbd] is closed, current opening sessions 0
2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.LaunchEngine: Processing myuser's query[2af47c76-a3a6-42fe-bd34-4c3aea3933f2]: FINISHED_STAT
E -> CLOSED_STATE, time taken: 34.601 seconds
2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=2af47c76-a3a6-42fe-bd34-4c3aea3933f2 opType=LaunchEngi
ne state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.537 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [5d12d959-c7a6-4dfc-8c18-6a08baf
63fbd]
2025-08-01 10:54:25.523 INFO KyuubiSessionManager-timeout-checker: Thread-63 org.apache.kyuubi.session.KyuubiSessionManager: Checking sessions timeout, current count: 0

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[pan3793]

It's by design, in this case, you will get an error report from your BeeLine client.

[sohurdc]

BeeLine client can get an error, But the existing job monitoring is all based on the YARN API. Is there any plan to support correcting job status in the future? Or is there a parameter can correcting job status ? You're very much appreciated.

[Echan0808]

I encountered the same issue under the sharelevel of connection. The JDBC client can receive the error, but the YARN app status shows as successful, which is somewhat troublesome for auditing. It seems likely that this is caused by the Kyuubi engine layer manually calling SparkContext.stop().