Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] Optimize cleanupStorageSelectionCache method in LocalStorageManager. #1209

Closed
3 tasks done
zhuyaogai opened this issue Sep 24, 2023 · 1 comment · Fixed by #1210
Closed
3 tasks done

[Improvement] Optimize cleanupStorageSelectionCache method in LocalStorageManager. #1209

zhuyaogai opened this issue Sep 24, 2023 · 1 comment · Fixed by #1210

Comments

@zhuyaogai
Copy link
Contributor

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

Assuming there are 500 running APPs, each APP has an average of 10 shuffleIds, and each shuffleId has 1000 partitions, then the size of partitionsOfStorage in LocalStorageManager is about 5000000. The cleanupStorageSelectionCache method can take 500ms on average (sometimes it took a longer time when the size of partitionsOfStorage is larger). So it should be optimized.

Also assuming we complete 10,000 APPs in a day, if we use the AppPurgeEvent event to release resources, it would take approximately 10,000 * 500ms = 5000s. But if we use the ShufflePurgeEvent event to release resources, then it would take 10,000 * 500ms * 10 = 50,000s, which is quite crazy.

How should we improve?

  1. Optimize cleanupStorageSelectionCache method in LocalStorageManager.
  2. Fix bug when release resource using ShufflePurgeEvent event.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zhuyaogai zhuyaogai mentioned this issue Sep 24, 2023
Merged
zuston pushed a commit that referenced this issue Sep 25, 2023
@zhuyaogai
[#1209] improvement(server): Speed up cleanupStorageSelectionCache me…
b4adaa5 
…thod in LocalStorageManager. (#1210)

### What changes were proposed in this pull request?

Optimize cleanupStorageSelectionCache method in LocalStorageManager.

### Why are the changes needed?

Fix: #1209

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?
Existing UTs.
@xumanbu
Copy link
Contributor

xumanbu commented Oct 31, 2023

Good Catch. I find the same problom.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@xumanbu @zhuyaogai