HADOOP-16202. openFile()

Name and document standard options; move to visible class definition.

Document better in filesystem spec, including
examples of API and MapReduce use

Change-Id: Ia593a67babcad63c13dda65b62287683a678a266

Author: steveloughran

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

@@ -4536,7 +4536,7 @@ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
       final OpenFileParameters parameters) throws IOException {
     AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
         parameters.getMandatoryKeys(),
-        OpenFileParameters.STANDARD_OPTIONS, "");
+        OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS, "");
     CompletableFuture<FSDataInputStream> result = new CompletableFuture<>();
     try {
       result.complete(open(pathHandle, parameters.getBufferSize()));

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FutureDataInputStreamBuilder.java

@@ -17,6 +17,7 @@
  */
 package org.apache.hadoop.fs;
 
+import javax.annotation.Nullable;
 import java.io.IOException;
 import java.util.concurrent.CompletableFuture;
 
@@ -27,14 +28,14 @@
  * Builder for input streams and subclasses whose return value is
  * actually a completable future: this allows for better asynchronous
  * operation.
- *
+ * <p></p>
  * To be more generic, {@link #opt(String, int)} and {@link #must(String, int)}
  * variants provide implementation-agnostic way to customize the builder.
  * Each FS-specific builder implementation can interpret the FS-specific
  * options accordingly, for example:
- *
+ * <p></p>
  * If the option is not related to the file system, the option will be ignored.
- * If the option is must, but not supported by the file system, a
+ * If the option is must, but not supported/known by the file system, an
  * {@link IllegalArgumentException} will be thrown.
  *
  */
@@ -51,10 +52,11 @@ CompletableFuture<FSDataInputStream> build()
   /**
    * A FileStatus may be provided to the open request.
    * It is up to the implementation whether to use this or not.
-   * @param status status.
+   * @param status status: may be null
    * @return the builder.
    */
-  default FutureDataInputStreamBuilder withFileStatus(FileStatus status) {
+  default FutureDataInputStreamBuilder withFileStatus(
+      @Nullable FileStatus status) {
     return this;
   }
 

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/OpenFileOptions.java

@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * The standard openfile options.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class OpenFileOptions {
+
+  /**
+   * Prefix for all standard filesystem options: {@value}.
+   */
+  public static final String FILESYSTEM_OPTION = "fs.option.";
+
+  /**
+   * Prefix for all openFile options: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE =
+      FILESYSTEM_OPTION + "openfile.";
+
+  /**
+   * OpenFile option for seek policies: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_LENGTH =
+      FS_OPTION_OPENFILE + "length";
+
+  /**
+   * OpenFile option for seek policies: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_FADVISE =
+      FS_OPTION_OPENFILE + "fadvise";
+
+  /**
+   * fadvise policy: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_FADVISE_NORMAL =
+      "normal";
+
+  /**
+   * fadvise policy: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_FADVISE_SEQUENTIAL =
+      "sequential";
+
+  /**
+   * fadvise policy: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_FADVISE_RANDOM =
+      "random";
+
+  /**
+   * fadvise policy: {@value}.
+   */
+  public static final String FS_OPTION_OPENFILE_FADVISE_ADAPTIVE =
+      "adaptive";
+
+  /**
+   * Set of standard options which openfile implementations
+   * MUST recognize, even if they ignore the actual values.
+   */
+  public static final Set<String> FS_OPTION_OPENFILE_STANDARD_OPTIONS =
+      Stream.of(
+          FS_OPTION_OPENFILE_FADVISE,
+          FS_OPTION_OPENFILE_LENGTH)
+          .collect(Collectors.toSet());
+
+}

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureDataInputStreamBuilderImpl.java

@@ -19,6 +19,7 @@
 package org.apache.hadoop.fs.impl;
 
 import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 import java.io.IOException;
 import java.util.concurrent.CompletableFuture;
 
@@ -40,16 +41,15 @@
  * Builder for input streams and subclasses whose return value is
  * actually a completable future: this allows for better asynchronous
  * operation.
- *
+ * <p></p>
  * To be more generic, {@link #opt(String, int)} and {@link #must(String, int)}
  * variants provide implementation-agnostic way to customize the builder.
  * Each FS-specific builder implementation can interpret the FS-specific
  * options accordingly, for example:
- *
+ * <p></p>
  * If the option is not related to the file system, the option will be ignored.
- * If the option is must, but not supported by the file system, a
+ * If the option is must, but not supported/known by the file system, an
  * {@link IllegalArgumentException} will be thrown.
- *
  */
 @InterfaceAudience.Public
 @InterfaceStability.Unstable
@@ -147,7 +147,8 @@ public FutureDataInputStreamBuilder getThisBuilder() {
   }
 
   @Override
-  public FutureDataInputStreamBuilder withFileStatus(FileStatus st) {
+  public FutureDataInputStreamBuilder withFileStatus(
+      @Nullable FileStatus st) {
     this.status = st;
     return this;
   }

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java

@@ -106,47 +106,4 @@ public FileStatus getStatus() {
     return status;
   }
 
-
-  /**
-   * OpenFile option for seek policies: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_FADVISE =
-      "fs.opt.openfile.fadvise";
-
-  /**
-   * fadvise policy: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_FADVISE_NORMAL = "normal";
-
-  /**
-   * fadvise policy: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_FADVISE_SEQUENTIAL = "sequential";
-
-  /**
-   * fadvise policy: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_FADVISE_RANDOM = "random";
-
-  /**
-   * fadvise policy: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_FADVISE_ADAPTIVE = "adaptive";
-
-  /**
-   * OpenFile option for seek policies: {@value}.
-   */
-  public static final String FS_OPT_OPENFILE_LENGTH =
-      "fs.opt.openfile.length";
-
-  /**
-   * Set of standard options.
-   */
-  public static final Set<String> STANDARD_OPTIONS =
-      Stream.of(
-          FS_OPT_OPENFILE_FADVISE,
-          FS_OPT_OPENFILE_LENGTH)
-          .collect(Collectors.toSet());
-
-
 }

hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md

@@ -728,97 +728,11 @@ exists in the metadata, but no copies of any its blocks can be located;
 
 ### `FSDataInputStreamBuilder openFile(Path path)`
 
-Creates a [`FSDataInputStreamBuilder`](fsdatainputstreambuilder.html)
-to construct a operation to open the file at `path` for reading.
+See [openFile()](openfile.html).
 
-When `build()` is invoked on the returned `FSDataInputStreamBuilder` instance,
-the builder parameters are verified and
-`openFileWithOptions(Path, OpenFileParameters)` invoked.
-
-This (protected) operation returns a `CompletableFuture<FSDataInputStream>`
-which, when its `get()` method is called, either returns an input
-stream of the contents of opened file, or raises an exception.
-
-The base implementation of the `openFileWithOptions(PathHandle, OpenFileParameters)`
-ultimately invokes `open(Path, int)`.
-
-Thus the chain `openFile(path).build().get()` has the same preconditions
-and postconditions as `open(Path p, int bufferSize)`
-
-However, there is one difference which implementations are free to
-take advantage of: 
-
-The returned stream MAY implement a lazy open where file non-existence or
-access permission failures may not surface until the first `read()` of the
-actual data.
-
-The `openFile()` operation may check the state of the filesystem during its
-invocation, but as the state of the filesystem may change betwen this call and
-the actual `build()` and `get()` operations, this file-specific
-preconditions (file exists, file is readable, etc) MUST NOT be checked here.
-
-FileSystem implementations which do not implement `open(Path, int)`
-MAY postpone raising an `UnsupportedOperationException` until either the
-`FSDataInputStreamBuilder.build()` or the subsequent `get()` call,
-else they MAY fail fast in the `openFile()` call.
-
-### Implementors notes
-
-The base implementation of `openFileWithOptions()` actually executes
-the `open(path)` operation synchronously, yet still returns the result
-or any failures in the `CompletableFuture<>`, so as to ensure that users
-code expecting this.
-
-Any filesystem where the time to open a file may be significant SHOULD
-execute it asynchronously by submitting the operation in some executor/thread
-pool. This is particularly recommended for object stores and other filesystems
-likely to be accessed over long-haul connections.
-
-Arbitrary filesystem-specific options MAY be supported; these MUST
-be prefixed with either the filesystem schema, e.g. `hdfs.`
-or in the "fs.SCHEMA" format as normal configuration settings `fs.hdfs`). The
-latter style allows the same configuration option to be used for both
-filesystem configuration and file-specific configuration.
-
-It SHOULD be possible to always open a file without specifying any options,
-so as to present a consistent model to users. However, an implementation MAY
-opt to require one or more mandatory options to be set.
-
-The returned stream may perform "lazy" evaluation of file access. This is
-relevant for object stores where the probes for existence are expensive, and,
-even with an asynchronous open, may be considered needless.
- 
 ### `FSDataInputStreamBuilder openFile(PathHandle)`
 
-Creates a `FSDataInputStreamBuilder` to build an operation to open a file.
-Creates a [`FSDataInputStreamBuilder`](fsdatainputstreambuilder.html)
-to construct a operation to open the file identified by the given `PathHandle` for reading.
-
-When `build()` is invoked on the returned `FSDataInputStreamBuilder` instance,
-the builder parameters are verified and
-`openFileWithOptions(PathHandle, OpenFileParameters)` invoked.
-
-This (protected) operation returns a `CompletableFuture<FSDataInputStream>`
-which, when its `get()` method is called, either returns an input
-stream of the contents of opened file, or raises an exception.
-
-The base implementation of the `openFileWithOptions(PathHandle, OpenFileParameters)` method
-returns a future which invokes `open(Path, int)`.
-
-Thus the chain `openFile(pathhandle).build().get()` has the same preconditions
-and postconditions as `open(Pathhandle, int)`
-
-As with `FSDataInputStreamBuilder openFile(PathHandle)`, the `openFile()`
-call must not be where path-specific preconditions are checked -that
-is postponed to the `build()` and `get()` calls.
-
-FileSystem implementations which do not implement `open(PathHandle handle, int bufferSize)`
-MAY postpone raising an `UnsupportedOperationException` until either the
-`FSDataInputStreamBuilder.build()` or the subsequent `get()` call,
-else they MAY fail fast in the `openFile()` call.
-
-The base implementation raises this exception in the `build()` operation;
-other implementations SHOULD copy this.
+See [openFile()](openfile.html).
 
 ### `PathHandle getPathHandle(FileStatus stat, HandleOpt... options)`
 

hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md

@@ -38,3 +38,5 @@ HDFS as these are commonly expected by Hadoop client applications.
 2. [Testing with the Filesystem specification](testing.html)
 2. [Extending the specification and its tests](extending.html)
 1. [Uploading a file using Multiple Parts](multipartuploader.html)
+1. [openFile()](openfile.html).
+