Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list #4623

Closed
3 of 4 tasks
zwangsheng opened this issue Mar 27, 2023 · 0 comments
Closed
3 of 4 tasks

[Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list #4623

zwangsheng opened this issue Mar 27, 2023 · 0 comments

Comments

@zwangsheng
Copy link
Contributor

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

KubernetesApplicationOperation makes too many requests to the kubernetes api server (in list form) when a large number of tasks are submitted at the same time, causing some stress.

How should we improve?

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/SharedInformerExample.java

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.
@zwangsheng zwangsheng mentioned this issue Mar 28, 2023
Closed
4 tasks
pan3793 added a commit that referenced this issue Mar 31, 2023
@zwangsheng @pan3793
[KYUUBI #4623][K8S] KubernetesApplicationOperation uses Informer inst…
3ab8733 
…ead of list

Close #4623

To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state.

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] Run CI

Closes #4625 from zwangsheng/KYUUBI_4623.

Closes #4623

a415bef [Cheng Pan] nit
136d0db [Cheng Pan] 171
b5d3c23 [Cheng Pan] re-generate conf
bf14ad8 [Cheng Pan] nit
9ee7e04 [Cheng Pan] nit
301162e [Cheng Pan] nit
1d42692 [Cheng Pan] nit
b95d7a6 [Cheng Pan] improve
cc8d2c7 [zwangsheng] fix comments
d017baf [zwangsheng] Set resycn 0
28f9a70 [zwangsheng] Reorder func & slow get app info
22d9c16 [zwangsheng] fix setting
8e09403 [zwangsheng] fix comments
10965d3 [zwangsheng] Rename fileter function => isSparkEnginePod
b026771 [zwangsheng] rename
78c9fdb [zwangsheng] fix comments
6d31f70 [zwangsheng] Fix IT Test
f43bba2 [zwangsheng] fix
17e4f55 [zwangsheng] debug
be8da79 [zwangsheng] debug
0db45a5 [zwangsheng] retest
a93786a [zwangsheng] Fix style
652ee83 [zwangsheng] Add Setting & Debug
4add7e4 [zwangsheng] improve
1f43412 [zwangsheng] remove unused import
35acd61 [zwangsheng] fix compile
05dfc59 [zwangsheng] [KYUUBI #4623][Improvement][K8S] Remove cached app info when out of time
4ab530e [zwangsheng] [KYUUBI #4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit to pan3793/kyuubi that referenced this issue Mar 31, 2023
@pan3793
[KYUUBI apache#4623][FOLLOUP] Remove unused import
de03bb5
pan3793 added a commit that referenced this issue Mar 31, 2023
@pan3793
[KYUUBI #4623][FOLLOWUP] Remove unused import
4e9e647 
### _Why are the changes needed?_

```
Error: ] /home/runner/work/kyuubi/kyuubi/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/spark/SparkOnKubernetesTestsSuite.scala:37: Unused import
Error: [ERROR] one error found
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4645 from pan3793/followup.

Closes #4623

de03bb5 [Cheng Pan] [KYUUBI #4623][FOLLOUP] Remove unused import

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit that referenced this issue Mar 31, 2023
@pan3793
[KYUUBI #4623][FOLLOWUP] Remove unused import
7358da9 
### _Why are the changes needed?_

```
Error: ] /home/runner/work/kyuubi/kyuubi/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/spark/SparkOnKubernetesTestsSuite.scala:37: Unused import
Error: [ERROR] one error found
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4645 from pan3793/followup.

Closes #4623

de03bb5 [Cheng Pan] [KYUUBI #4623][FOLLOUP] Remove unused import

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 4e9e647)
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit that referenced this issue Apr 4, 2023
@pan3793
[KYUUBI #4623][FOLLOWUP] Fix malformed docs
9c52cc2
pan3793 added a commit to pan3793/kyuubi that referenced this issue May 12, 2023
@zwangsheng @pan3793
[KYUUBI apache#4623][K8S] KubernetesApplicationOperation uses Informe…
1126b10 
…r instead of list

Close apache#4623

To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state.

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] Run CI

Closes apache#4625 from zwangsheng/KYUUBI_4623.

Closes apache#4623

a415bef [Cheng Pan] nit
136d0db [Cheng Pan] 171
b5d3c23 [Cheng Pan] re-generate conf
bf14ad8 [Cheng Pan] nit
9ee7e04 [Cheng Pan] nit
301162e [Cheng Pan] nit
1d42692 [Cheng Pan] nit
b95d7a6 [Cheng Pan] improve
cc8d2c7 [zwangsheng] fix comments
d017baf [zwangsheng] Set resycn 0
28f9a70 [zwangsheng] Reorder func & slow get app info
22d9c16 [zwangsheng] fix setting
8e09403 [zwangsheng] fix comments
10965d3 [zwangsheng] Rename fileter function => isSparkEnginePod
b026771 [zwangsheng] rename
78c9fdb [zwangsheng] fix comments
6d31f70 [zwangsheng] Fix IT Test
f43bba2 [zwangsheng] fix
17e4f55 [zwangsheng] debug
be8da79 [zwangsheng] debug
0db45a5 [zwangsheng] retest
a93786a [zwangsheng] Fix style
652ee83 [zwangsheng] Add Setting & Debug
4add7e4 [zwangsheng] improve
1f43412 [zwangsheng] remove unused import
35acd61 [zwangsheng] fix compile
05dfc59 [zwangsheng] [KYUUBI apache#4623][Improvement][K8S] Remove cached app info when out of time
4ab530e [zwangsheng] [KYUUBI apache#4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #4623][K8S] KubernetesApplicationOperation uses Informer instead of list
1 participant
@zwangsheng