HADOOP-19256. steve's pr of conditional writes #7362
+1,615
−103
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
force-pushed
the
s3/HADOOP-19256-s3-conditional-writes
branch
from
82aa6e1 to
d04aa2e
Compare
|
Tested: S3 london, |
|
@saikatroy038 the two commits here are ready to be cherrypicked. |
steveloughran
force-pushed
the
s3/HADOOP-19256-s3-conditional-writes
branch
3 times, most recently
from
63c30eb to
222a171
Compare
steveloughran
force-pushed
the
s3/HADOOP-19256-s3-conditional-writes
branch
from
147fed7 to
fe59fc0
Compare
+ hadoop common and s3a create file options wired up to s3a create file builder, and nowhere else. Change-Id: Ieaafa9ecdfd68306deb41ecf40c416374a310859
…art files createFile() options are passed down all the way to object creation * fs.option.create.conditional.overwrite: sets the header. MUST be in close(). After all, create(overwrite=false) is eager. * fs.option.create.conditional.overwrite.etag requests etag writes. MAY be in create(); may be in close(). * fs.option.create.content.type for mime type. * fs.option.create.in.close to ask for create in close, as fs capability. (maybe make this an fs path capability only?) * fs.s3a.create.multipart : allows tests to create small multipart files Javadocs of the fs.option.create try to define semantics; will need strict fs specification soon. Bool params come down in an enumset of flags; I'm going to do that in more code as it is more flexible over time than many booleans. - etag values are passed down but not wired up/tested - content type flag exists but is ignored New WriteObjectFlags enum is in new package o.a.h.fs.s3a.write; to match the streams package in another ongoing PR. Goal, as we maintain things, all code related to writing can go in here. also: options set by WriteObjectFlags return true in hasCapability() of the stream Change-Id: I301abd7397accbd278d05f42f858223ba1349fc8
New service under S3AStoreImpl: StoreConfigurationService Change-Id: I99c8305574492c05170274dcc363bfba4857981b
S3AFS supports enable/disable of conditional create, and ability to turn it on automatically. fs.s3a.conditional.create.enabled: option to enable (default) or disable conditional creation fs.s3a.conditional.create.files: option to enable or disable (default) using conditional create when writing any file with overwrite = false. Test setups do not (yet) skip test suites if fs.s3a.conditional.create.files is false; they should and it needs to be updated in the third-party stores section Change-Id: I96a923b84cf29efaaae7ced41bfb72f8944be1ca
steveloughran
force-pushed
the
s3/HADOOP-19256-s3-conditional-writes
branch
from
fe59fc0 to
6a59bb1
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
4 tasks
Another iteration; * pulled out *all* the stuff related to the notion of a configuration service -needs to be independent PR * "simplified" logic as to when to use conditional create for a normal create() call, even without any explicit request. * doc updates Change-Id: I6a5770d4d85197f4f9338900cc21f9bca8b7c704
- testIfMatchCreateFileWithoutOverwrite, testIfMatchCreateFileWithoutOverwriteWithPerformanceFlag - with the new design, overwrite is ignored if etag is passed
|
Forced push of saikat's final PR and a checkstyle fix. final test run and review underway...If I'm happy this will be what I merge in |
|
I am +1 pending yetus; while I've done some of this most of the work is from others |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
ok, ready to merge. doing a local flattening of the PR as with the merges and things it's got complex. then will push up the final single patch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#7329 with some changes