HADOOP-18373. IOStatisticsContext tuning #4705
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested against Errors: |
|
🎊 +1 overall
This message was automatically generated. |
|
For
Tried with both buckets, with and without ACL enabled. This doesn't seem relevant to the patch but wondering what I am missing in the configs specifically for |
virajjasani
force-pushed
the
HADOOP-18373-trunk
branch
from
8c5d530
to
127d230
Compare
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Aug 5, 2022
| @@ -178,6 +178,12 @@ | |||
| <value>true</value> | |||
| </property> | |||
|
|
|||
| <!-- Enable IOStatisticsContext support for Thread level. --> | |||
There was a problem hiding this comment.
mixed feelings here. hive team and others who serialize configs don't like us overfilling the xml file with defaults
There was a problem hiding this comment.
They might not have issue with test resource xml changes right?
There was a problem hiding this comment.
no issues there at all
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AIOStatisticsContext.java
Outdated
Show resolved
Hide resolved
...project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/IOStatisticsContext.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
never seen that; docs say "AWS access key ID provided does not exist in our records." it might be that the arn of the token you are asking for doesn't exist, or that you don't have permissions to create sessions for it and it is failing |
|
merged to trunk; testing branch-3.3 and will push up if all is good |
asfgit
pushed a commit
that referenced
this pull request
Aug 8, 2022
The name of the option to enable/disable thread level statistics is "fs.iostatistics.thread.level.enabled"; There is also an enabled() probe in IOStatisticsContext which can be used to see if the thread level statistics is active. Contributed by Viraj Jasani
Thanks @steveloughran. Here is what I did: created role, provided policy, created user, provided the same policy. Updated role's trust relationship to allow user to perform assume-role on the role. Perhaps the user (who did the assume-role) doesn't have some specific permission? |
|
I was able to figure out the issue, please ignore above comment. |
|
what was it? test running as other user? fwiw i use a very restricted user/role for my s3 tests, so if that key ever leaked, it would limit the damage to accessing my test s3 buckets, assume one role, etc. not even start an EC2 vm |
Basically for this test, I manually did the
Ah yes, this is definitely very good suggestion, thanks! |
|
yeah, you can't ask for session creds with session creds cloudstore has a command to ask for session credentials with your full s3a secrets, prints as env vars, properties, xml handy for getting some secrets into a vm for a while |
Thank you for sharing, Steve! |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
The name of the option to enable/disable thread level statistics is "fs.iostatistics.thread.level.enabled"; There is also an enabled() probe in IOStatisticsContext which can be used to see if the thread level statistics is active. Contributed by Viraj Jasani
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Tuning of the IOStatisticsContext code
change property name to "fs.iostatistics...." for consistent naming
Edit core-site.xml to always collect context iOStatistics