HDFS-16949 Update ReadTransferRate to ReadLatencyPerGB for effectiv… #5479
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
…e percentile metrics
| q.add(readTransferRate); | ||
| public void addReadLatencyPerGB(long readLatencyPerGB) { | ||
| this.readLatencyPerGB.add(readLatencyPerGB); | ||
| for (MutableQuantiles q : readLatencyPerGBQuantiles) { |
There was a problem hiding this comment.
Why don't we modify these quantiles rather than the metric?
I think we have the metric we want, but we want the lowest vs highest percentile (ts clear the current percentile is lower=better).
|
💔 -1 overall
This message was automatically generated. |
|
Closing as a different approach was merged in PR #5495 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…e percentile metrics
Description of PR
PR #5397 added ReadTransferRate quantiles to calculate the rate which data is read per unit of time.
With percentiles the values are sorted in ascending order and hence for the transfer rate p90 gives us the value where 90 percent rates are lower (worse), p99 gives us the value where 99 percent values are lower (worse).
Note that value(p90) < p(99) thus p99 is a better transfer rate as compared to p90.
However as the percentile increases the value should become worse in order to know how good our system is.
Hence instead of calculating the data read transfer rate, we should calculate it's inverse. We will instead calculate the time taken for a GB of data to be read. ( seconds / GB )
After this the p90 value will give us 90 percentage of total values where the time taken is less than value(p90), similarly for p99 and others.
Also p(90) < p(99) and here p(99) will become a worse value (taking more time each byte) as compared to p(90)
How was this patch tested?
Updated Unit Tests.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?