-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDFS-16446. Consider ioutils of disk when choosing volume #3960
base: trunk
Are you sure you want to change the base?
Conversation
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some quick comments. I'm still trying to understand how the disk io stats get considered. Some documentation would be helpful.
Instead of being part of AvailableSpaceVolumeChoosingPolicy, would it make sense to become a new volume choosing policy?
...op-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
Outdated
Show resolved
Hide resolved
...ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskIOUtilManager.java
Show resolved
Hide resolved
...ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskIOUtilManager.java
Outdated
Show resolved
Hide resolved
...ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskIOUtilManager.java
Show resolved
Hide resolved
...ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskIOUtilManager.java
Outdated
Show resolved
Hide resolved
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
Outdated
Show resolved
Hide resolved
@ayushtkn fyi |
|
Thanks @jojochuang for your comments, I will add documentation and update the code. When doing the POC testing, we define a new policy, named So we change AvailableSpaceVolumeChoosingPolicy, under the same disk usage, preferred ioutils smaller disks, filtering the busy disks. What you said about offering a new policy is a good idea. |
💔 -1 overall
This message was automatically generated. |
4190ea9
to
754f133
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Co-authored-by: liuhongtong <hongtongliu@126.com> Co-authored-by: hfutatzhanghb <1036798979@qq.com>
💔 -1 overall
This message was automatically generated. |
Hi @jojochuang @ayushtkn @tasanuma , I added a document for this. Could you please take a look at this? Thanks a lot. The failed unit tests seem unrelated to this change. |
Hi, @tomscut , we also face slow nodes caused by io, but in most cases it is slow to read data because the frequency of reading in the cluster far exceeds that of writing. We have counted many cases, when reading slowly, the iowait is basically high, but only one or a few disks are full of ioutil (12 disks per dn). Therefore, I have the following points to discuss.
|
@whbing Do you have any strategies to deal with the above two points? |
Hi, @whbing. Your first suggestion is good. But sorry I've some other opinions, VolumeChoosingPolicy is only relevant during the write operation, determining which disk on a DataNode should store the data block. When reading files from HDFS, the focus is on selecting the appropriate DataNode that contains the required data blocks, not on selecting specific disks within a DataNode. Sorry if I'm wrong. |
JIRA: HDFS-16446.
Consider ioutils of disk when choosing volume to avoid busy disks.
Document: https://docs.google.com/document/d/1Ko1J7shz8hVLnNACT6PKVQ_leIHf_YaIFA2s3yJMZHQ/edit?usp=sharing
Architecture is as follows: