HADOOP-17597. Optionally downgrade on S3A Syncable calls #2801
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Testing: new tests all happy. Now regression testing with s3 london |
|
💔 -1 overall
This message was automatically generated. |
|
Test s3 london with test options Failures |
steveloughran
force-pushed
the
s3/HADOOP-17597-s3a-syncable
branch
from
91c7e74 to
9cbca3e
Compare
|
💔 -1 overall
This message was automatically generated. |
mukund-thakur
approved these changes
Apr 1, 2021
|
|
||
| These recommendations _apply to all filesystems_. | ||
|
|
||
| To downgrade the S3A connector to simplying warning of the use of |
There was a problem hiding this comment.
nit : typo simply warn the?
| } | ||
|
|
||
| @Test | ||
| public void testHFlushDowngrade() throws Throwable { |
| } | ||
|
|
||
| @Test | ||
| public void testHSyncDowngrade() throws Throwable { |
There was a problem hiding this comment.
nit: add describe test
| } | ||
|
|
||
| /** | ||
| * When configured to downgrade, the stream downgrades on. |
There was a problem hiding this comment.
nit: no fullstop in the end.
|
thanks, will update the PR with the comments. |
|
build failed. Will rebase and resubmit |
Adds a new option fs.s3a.downgrade.syncable.exceptions Which converts calls to Syncable hsync/hflush on S3A output streams to * log once at warn (for entire process life, not just the stream) * increment IOStats with the relevant operation counter With the downgrade not enabled (default) * IOStats also incremented * The UnsupportedOperationException current raised includes a link to this JIRA Change-Id: I650a8da55cf32725d4ac9c9db3f466cc427669fd
Checkstyle and troubleshooting Change-Id: Ibb11fe85531a07acc9cf66d2fc941103784a3d27
Change-Id: Ibb0fd0c47d6da129c9ab77d45011e2b9f8d4bab1
steveloughran
force-pushed
the
s3/HADOOP-17597-s3a-syncable
branch
from
2a46669 to
9643ec0
Compare
|
💔 -1 overall
This message was automatically generated. |
|
test failure was in hadoop common and kerberos auth related, ignoring. Merging |
asfgit
pushed a commit
that referenced
this pull request
Apr 24, 2021
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls to raise an exception. Adds a new option fs.s3a.downgrade.syncable.exceptions When true, calls to Syncable hsync/hflush on S3A output streams will log once at warn (for entire process life, not just the stream), then increment IOStats with the relevant operation counter With the downgrade option false (default) * IOStats are incremented * The UnsupportedOperationException current raised includes a link to the JIRA. Contributed by Steve Loughran. Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls to raise an exception. Adds a new option fs.s3a.downgrade.syncable.exceptions When true, calls to Syncable hsync/hflush on S3A output streams will log once at warn (for entire process life, not just the stream), then increment IOStats with the relevant operation counter With the downgrade option false (default) * IOStats are incremented * The UnsupportedOperationException current raised includes a link to the JIRA. Contributed by Steve Loughran.
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
…pache#2801) Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls to raise an exception. Adds a new option fs.s3a.downgrade.syncable.exceptions When true, calls to Syncable hsync/hflush on S3A output streams will log once at warn (for entire process life, not just the stream), then increment IOStats with the relevant operation counter With the downgrade option false (default) * IOStats are incremented * The UnsupportedOperationException current raised includes a link to the JIRA. Contributed by Steve Loughran. Change-Id: I650a8da55cf32725d4ac9c9db3f466cc427669fd
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new option fs.s3a.downgrade.syncable.exceptions
Which converts calls to Syncable hsync/hflush on S3A output streams to
With the downgrade not enabled (default)
JIRA