Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilize Unified Sorter #1148

Open
5 tasks
liuzix opened this issue Dec 1, 2020 · 0 comments
Open
5 tasks

Stabilize Unified Sorter #1148

liuzix opened this issue Dec 1, 2020 · 0 comments
Assignees
@liuzix
Labels
component/puller Puller component. status/need-discussion Issue that needs to be discussed to confirm priority, milestone, plan and task breakdown.

Comments

@liuzix
Copy link
Contributor

liuzix commented Dec 1, 2020 •
edited by amyangfei
Loading

Feature Request

Is your feature request related to a problem? Please describe:

  • Unified Sorter needs to be stabilized so that users do not face unnecessary OOMs anymore.

Describe the feature you'd like:

  • reduce Goroutines.
  • suffix to temporary files created by the Sorter. P0
  • file checksum.
  • better file recycling.
  • encryption at rest.

Teachability, Documentation, Adoption, Migration Strategy:

  • The number of Goroutines used should be reduced so that (1) overall performance is improved, and (2) race detector can be safely enabled.
  • Recycling mechanism should be optimized to avoid calling truncate. We should aim to reduce inode IO as much as possible.
@liuzix liuzix added the component/puller Puller component. label Dec 1, 2020
@liuzix liuzix self-assigned this Dec 1, 2020
@amyangfei amyangfei added the status/need-discussion Issue that needs to be discussed to confirm priority, milestone, plan and task breakdown. label Dec 1, 2020
@amyangfei amyangfei mentioned this issue Dec 1, 2020
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

Labels
component/puller Puller component. status/need-discussion Issue that needs to be discussed to confirm priority, milestone, plan and task breakdown.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amyangfei @liuzix