-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-17928. Syncable: S3A to warn and downgrade #3585
HADOOP-17928. Syncable: S3A to warn and downgrade #3585
Conversation
This switches the default behavior of S3A output streams to warning that Syncable.hsync() or hflush() have been called; it's not considered an error unless the defaults are overridden. This avoids breaking applications which call the APIs, at the risk of people trying to use S3 as a safe store of streamed data (HBase WALs, audit logs etc). Contributed by Steve Loughran. Change-Id: Id7f8fc65a32a5ab664ebbd68d92754d4c38e29d8
Testing in progress. FWIW, internally we had the core-default setting up for a while, because various obscure apps (ranger logs &c) were calling hflush and hsync, and while "stop it" is the correct long-term fix, that relies co-ordination across OSS release schedules. Which as we know, doesn't exist. |
Change-Id: Ic9fb5026f13904bc4921946a75b9d807b2db326d
Test result, all good except for the buffer incomplete issue which happens locally for me
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good. Pending that blanks fix caught by Yetus.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! 👍
Thank you for pinging me, @sunchao . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM.
thx, will fix the eof and apply |
Change-Id: I61441b01f1aee342f3e59da7ef107b1dfc023e1d
ok, as soon as yetus is done i will merge |
🎊 +1 overall
This message was automatically generated. |
This switches the default behavior of S3A output streams to warning that Syncable.hsync() or hflush() have been called; it's not considered an error unless the defaults are overridden. This avoids breaking applications which call the APIs, at the risk of people trying to use S3 as a safe store of streamed data (HBase WALs, audit logs etc). Contributed by Steve Loughran. Change-Id: I0a02ec1e622343619f147f94158c18928a73a885
This switches the default behavior of S3A output streams to warning that Syncable.hsync() or hflush() have been called; it's not considered an error unless the defaults are overridden. This avoids breaking applications which call the APIs, at the risk of people trying to use S3 as a safe store of streamed data (HBase WALs, audit logs etc). Contributed by Steve Loughran.
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.
This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).
Contributed by Steve Loughran.
Change-Id: Id7f8fc65a32a5ab664ebbd68d92754d4c38e29d8
How was this patch tested?
New unit tests; cloud store test in progress
For code changes: