HADOOP-17271. S3A connector to support IOStatistics. #2580
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gabor has reported a test failure but I can't replicate it. I'm going to merge this in and then add a followup stabilisation patch for that and any other inevitable surprises -no CP to branch-3.3 until that is in |
This comment has been minimized.
This comment has been minimized.
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I77c0502f26f7ee95c96ca9d017358138e5fa20d0
steveloughran
force-pushed
the
s3/HADOOP-16830-HADOOP-17271-iostatistics-s3a
branch
from
3510257 to
e1d402e
Compare
took a few goes to work out the correct way to get the ambiguous getIOStatistics() references right. Change-Id: I80f44bf5d71eb7983e81cbc0e09d44af0a1e1da3
|
latest patches fix up the style/findbugs warnings. I've also replicated and fixed the test failure gabor reported. I am seeing a failure on huge uploads and putbyte counts, but I want to do that later just to get this one in by the end of 2020. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Again, this is yetus playing up; local builds and tests are fine. I am seeing an intermittent test failure on parallel runs of ITestHugeFile uploads, with a counter not incrementing, but this doesn't surface standalone. Will file and fix separately |
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Jan 13, 2021
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
asfgit
pushed a commit
that referenced
this pull request
Jan 14, 2021
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
steveloughran
deleted the
s3/HADOOP-16830-HADOOP-17271-iostatistics-s3a
branch
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
…che#2580) S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
deepakdamri
pushed a commit
to acceldata-io/hadoop
that referenced
this pull request
Jan 21, 2025
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
deepakdamri
pushed a commit
to acceldata-io/hadoop
that referenced
this pull request
Jan 21, 2025
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
deepakdamri
pushed a commit
to acceldata-io/hadoop
that referenced
this pull request
Jan 21, 2025
S3A connector to support the IOStatistics API of HADOOP-16830, This is a major rework of the S3A Statistics collection to * Embrace the IOStatistics APIs * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
successor to #2553
This is a major rework of the S3A Statistics collection to
collectors to interface/implementation classes in new packages.
S3AFileSystem, input and output streams, RemoteIterator instances
provided in list calls.
Regarding statistic collection, as well as all existing
statistics, the connector now records min/max/mean durations
of HTTP GET and HEAD requests, and those of LIST operations.
Contributed by Steve Loughran.