[TASK][MEDIUM] Engine adaptive pool size

[pan3793]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Basic knowledge of Scala programming language
• Familiar with the architect of Apache Kyuubi

Background and Goals

In kyuubi.engine.share.level non-CONNECTION mode, how many engines the Kyuubi server starts is determined by kyuubi.engine.pool.size.

The engine selection policy is determined by kyuubi.engine.pool.selectPolicy. The default is RANDOM, which means randomly selecting a number within the engine pool size range and checking whether the engine corresponding to this number has been created. If not, create one.

When kyuubi.engine.pool.size is a relatively large value, it may randomly create many engines.

In addition, kyuubi.session.engine.idle.timeout configures a relatively large idle time, so that once the engine is created, it will be slower to release the engine.

This selection strategy is a bit of a waste of resources, especially since it takes a while to start the spark cluster mode.

Implementation steps

We can expose some information about the engine load in the Engine. When the server selects the engine, it will decide whether to use a new engine or reuse the old engine based on the load.

Additional context

Original reporter is @cxzl25

Introduction of #6232

[beryllw]

I am interested in this and I expect to achieve it in a month, can i take this？

[lsm1]

Hi , I also interested in this task, can you assign it to me ? I expect to achieve it in a month Thanks.

[pan3793]

@Kwafoor @lsm1 thanks for your passion, I suggest you discuss offline first :)

[beryllw]

Hi , I also interested in this task, can you assign it to me ? I expect to achieve it in a month Thanks.

If you're still interested in this, I can try other tasks.

[lsm1]

Hi , I also interested in this task, can you assign it to me ? I expect to achieve it in a month Thanks.

If you're still interested in this, I can try other tasks.

Yes, I want to achieve it, thanks a lot

[pan3793]

@Kwafoor @lsm1 thanks to all of you for your passion, I assigned this task to @lsm1, looking forward to your excellent work