Commit 31c1fa1
[KYUUBI #6463] Release semaphore immediately after startup process exit
# 🔍 Description
## Issue References 🔗
The concurrency limit for the engine startup process is mainly used to avoid overload on the machine(or container) of the Kyuubi server, the current implementation holds startupProcessSemaphore until the session is established successfully. While for Spark on YARN cluster mode, some YARN queue resource insufficiency may block the subsequent Spark application submissions to other queues, significantly affecting the Kyuubi server's resource utilization.
## Describe Your Solution 🔧
We should immediately release the `startupProcessSemaphore` after the engine startup process exits (i.e., after the `spark-submit` process exits) as the load has already disappeared.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70%
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6463 from ic4y/master-p003.
Closes #6463
f7de68c [ic4y] Improve code quality
d8b0248 [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency Limit
Authored-by: ic4y <ic4y@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 95ed748)
Signed-off-by: Cheng Pan <chengpan@apache.org>1 parent 460e553 commit 31c1fa1
File tree
1 file changed
+4
-1
lines changed- kyuubi-server/src/main/scala/org/apache/kyuubi/engine
1 file changed
+4
-1
lines changedLines changed: 4 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
235 | | - | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
236 | 239 | | |
237 | 240 | | |
238 | 241 | | |
| |||
0 commit comments