Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] use the disk size obtained from periodic check to determine whether the disk can be written #1684

Closed
3 tasks done
xianjingfeng opened this issue May 9, 2024 · 0 comments · Fixed by #1685
Closed
3 tasks done

[Improvement] use the disk size obtained from periodic check to determine whether the disk can be written #1684

xianjingfeng opened this issue May 9, 2024 · 0 comments · Fixed by #1685

Comments

@xianjingfeng
Copy link
Member

xianjingfeng commented May 9, 2024 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

In our production cluster, we found that although the amount of shuffle data did not change much, the amount of writing to HDFS continued to increase every day. The main reason is that currently we determine whether the local disk is writable based on metadata.
Related to #1678 #1247

image image

How should we improve?

Use the disk size obtained from periodic check to determine whether the disk can be written.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@xianjingfeng xianjingfeng changed the title [Improvement] add a switch to skip determine whether it is writable by using the disk size in metaData [Improvement] use the disk size obtained from periodic check to determine whether the disk can be written May 9, 2024
@xianjingfeng xianjingfeng mentioned this issue May 9, 2024
Merged
xianjingfeng added a commit that referenced this issue May 11, 2024
@xianjingfeng
[#1684] fix(server): use the diskSize obtained from periodic check to…
40bd14b 
… determine whether is writable (#1685)

### What changes were proposed in this pull request?
Use the disk size obtained from periodic check to determine whether the disk can be written.

### Why are the changes needed?
The disk size obtained from metadata is unreliable.
Fix: #1684

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Existing UTs
xianjingfeng added a commit that referenced this issue May 11, 2024
@xianjingfeng
[#1684] fix(server): use the diskSize obtained from periodic check to…
416ecfa 
… determine whether is writable (#1685)

### What changes were proposed in this pull request?
Use the disk size obtained from periodic check to determine whether the disk can be written.

### Why are the changes needed?
The disk size obtained from metadata is unreliable.
Fix: #1684

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Existing UTs

(cherry picked from commit 40bd14b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@xianjingfeng