HADOOP-13327 Output Stream Specification. #1694
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
force-pushed
the
filesystem/HADOOP-13327-outputstream-spec-lean
branch
from
1431540 to
5237cd1
Compare
hadoop-yetus
reviewed
Nov 5, 2019
| ### HDFS and `OutputStream.close()` | ||
|
|
||
| HDFS does not immediately `sync()` the output of a written file to disk on | ||
| `OutputStream.close()` unless configured with `dfs.datanode.synconclose` |
hadoop-yetus
reviewed
Nov 5, 2019
| ``` | ||
|
|
||
| HDFS does not do this except when the write crosses a block boundary; to do | ||
| otherwise would overload the Namenode. Other stores MAY copy this behavior. |
hadoop-yetus
reviewed
Nov 5, 2019
| When an output stream in HDFS is closed; the newly written data is not immediately | ||
| written to disk unless HDFS is deployed with `dfs.datanode.synconclose` set to | ||
| true. Otherwise it is cached and written to disk later. | ||
|
|
hadoop-yetus
reviewed
Nov 5, 2019
| ### `StreamCapabilities` | ||
|
|
||
| Implementors of filesystem clients SHOULD implement the `StreamCapabilities` | ||
| interface and its `hasCapabilities()` method to to declare whether or not |
|
💔 -1 overall
This message was automatically generated. |
DadanielZ
reviewed
Dec 2, 2019
|
|
||
| /** | ||
| * Probe for an object having a capability; returns true | ||
| * iff the stream implements {@link StreamCapabilities} and its |
There was a problem hiding this comment.
typo? there are several iff in this doc.
There was a problem hiding this comment.
"if and only if"; CS term. I think I mention it somewhere -if not I should. It's used to be absolutely clear that if the condition is not met then the outcome is blocked
There was a problem hiding this comment.
I can cut it from the javadocs & leave in the markdown files, as they are the stricter bits of work
This PR removes the changes related to S3A output stream lifecycle, so only covers the specification of Syncable and ensures that StreamCapabilities passes all the way through to the final implementation classes. All streams which implement Syncable hsync/hflush declare this in their stream capabilities Change-Id: I82b16a8e0965f34eb0c42504da43e8fbeabcb68c
Change-Id: Id38cf27639215abdd0d8c3578ddf72ed7adca8c5
TODO "Could this be in a section about visibility and not in the model definition? Maybe later. "here's the model, here's how that model works with creation, here's how it works when reading/writing" flows much better and visibility should be in that third part." Change-Id: I61c89475a1ea72006524803f2a7dd9e40551d718
Review with more on 404 caching. Change-Id: Ib474a84e48556c6b76121427a026fa854b5bd9e0
steveloughran
force-pushed
the
filesystem/HADOOP-13327-outputstream-spec-lean
branch
from
5237cd1 to
c17cf2e
Compare
|
closing to rebase and resubmit |
|
rebased to #2102 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the changes related to S3A output stream lifecycle,
so only covers the specification of Syncable and ensures that StreamCapabilities
passes all the way through to the final implementation classes.
All streams which implement Syncable hsync/hflush declare this in their stream capabilities
Supercedes #575
Change-Id: I82b16a8e0965f34eb0c42504da43e8fbeabcb68c