Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Provide dedicated unregister app rpc interface #1476

Closed
2 of 3 tasks
zuston opened this issue Jan 23, 2024 · 4 comments · Fixed by #1510 or #1511
Closed
2 of 3 tasks

[FEATURE] Provide dedicated unregister app rpc interface #1476

zuston opened this issue Jan 23, 2024 · 4 comments · Fixed by #1510 or #1511
Assignees
Labels
good first issue Good for newcomers

Comments

@zuston
Copy link
Member

zuston commented Jan 23, 2024

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the feature

@Override
  public void unregisterShuffle(String appId) {
    if (appId == null) {
      return;
    }
    Map<Integer, Set<ShuffleServerInfo>> appServerMap = shuffleServerInfoMap.get(appId);
    if (appServerMap == null) {
      return;
    }
    appServerMap.keySet().forEach(shuffleId -> unregisterShuffle(appId, shuffleId));
  }

Motivation

I found the unregister app in shuffleManager stop method always is timeout when having too much stages. And there is no need to unregister shuffle one by one, why not introducing unregister app rpc method?

But I'm hesitant whether to reuse the unregisterShuffle rpc by making the shuffleId of protobuf optional to achieve the app purging

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston zuston added the good first issue Good for newcomers label Jan 23, 2024
@zuston zuston changed the title [FEATURE] Parallelize unregister shuffle for spark client [FEATURE] Provide dedicated unregister app rpc interface Feb 1, 2024
@zuston
Copy link
Member Author

zuston commented Feb 2, 2024

cc @xianjingfeng @jerqi . WDYT? Whether to introduce new rpc or reusing the existing

@xianjingfeng
Copy link
Member

xianjingfeng commented Feb 4, 2024

cc @xianjingfeng @jerqi . WDYT? Whether to introduce new rpc or reusing the existing

I think it is better to introduce new rpc interface.

@somelovelanguage
Copy link
Contributor

Kindly to ask if there is somebody working for this issue? If not, i'm glad to undertake it.

@zuston
Copy link
Member Author

zuston commented Feb 4, 2024

Kindly to ask if there is somebody working for this issue? If not, i'm glad to undertake it.

Feel free. Assigned

somelovelanguage added a commit to somelovelanguage/incubator-uniffle that referenced this issue Feb 6, 2024
Signed-off-by: Yiqi You <2810592075@qq.com>
zuston pushed a commit that referenced this issue Feb 8, 2024
…1510)

### What changes were proposed in this pull request?
Introduce dedicated unregisterApp rpc interface which is only called once when unregister shuffle

### Why are the changes needed?
Fix: [#1476](#1476)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT
zuston added a commit to zuston/incubator-uniffle that referenced this issue Feb 8, 2024
zuston added a commit that referenced this issue Feb 9, 2024
…1511)

### What changes were proposed in this pull request?

Provide dedicated unregister app rpc interface for rust based server

### Why are the changes needed?

Fix: #1476

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.
zuston pushed a commit to zuston/incubator-uniffle that referenced this issue Mar 19, 2024
…face (apache#1510)

### What changes were proposed in this pull request?
Introduce dedicated unregisterApp rpc interface which is only called once when unregister shuffle

### Why are the changes needed?
Fix: [apache#1476](apache#1476)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment