HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep" (#5689)

This 
1. changes the default value of fs.s3a.directory.marker.retention
   to "keep"
2. no longer prints a message when an S3A FS instances is
   instantiated with any option other than delete.

Switching to marker retention improves performance
on any S3 bucket as there are no needless marker DELETE requests
-leading to a reduction in write IOPS and and any delays waiting
for the DELETE call to finish.

There are *very* significant improvements on versioned buckets,
where tombstone markers slow down LIST operations: the more
tombstones there are, the worse query planning gets.

Having versioning enabled on production stores is the foundation
of any data protection strategy, so this has tangible benefits
in production.

It is *not* compatible with older hadoop releases; specifically
- Hadoop branch 2 < 2.10.2
- Any release of Hadoop 3.0.x and Hadoop 3.1.x
- Hadoop 3.2.0 and 3.2.1
- Hadoop 3.3.0
Incompatible releases have no problems reading data in stores
where markers are retained, but can get confused when deleting
or renaming directories.

If you are still using older versions to write to data, and cannot
yet upgrade, switch the option back to "delete"

Contributed by Steve Loughran

Author: steveloughran

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

@@ -1088,7 +1088,7 @@ private Constants() {
    * Default retention policy: {@value}.
    */
   public static final String DEFAULT_DIRECTORY_MARKER_POLICY =
-      DIRECTORY_MARKER_POLICY_DELETE;
+      DIRECTORY_MARKER_POLICY_KEEP;
 
 
   /**

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DirectoryPolicyImpl.java

@@ -186,11 +186,11 @@ public static DirectoryPolicy getDirectoryPolicy(
       policy = DELETE;
       break;
     case DIRECTORY_MARKER_POLICY_KEEP:
-      LOG.info("Directory markers will be kept");
+      LOG.debug("Directory markers will be kept");
       policy = KEEP;
       break;
     case DIRECTORY_MARKER_POLICY_AUTHORITATIVE:
-      LOG.info("Directory markers will be kept on authoritative"
+      LOG.debug("Directory markers will be kept on authoritative"
           + " paths");
       policy = new DirectoryPolicyImpl(MarkerPolicy.Authoritative,
           authoritativeness);

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md

@@ -792,7 +792,7 @@ Security
         Delegation token support is disabled
 
 Directory Markers
-        The directory marker policy is "delete"
+        The directory marker policy is "keep"
         Available Policies: delete, keep, authoritative
         Authoritative paths: fs.s3a.authoritative.path=```
 ```

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md

@@ -23,11 +23,10 @@
 
 ###  <a name="directory-marker-compatibility"></a> Directory Marker Compatibility
 
-1. This release can safely list/index/read S3 buckets where "empty directory"
-markers are retained.
-
-1. This release can be configured to retain these directory makers at the
-expense of being backwards incompatible.
+This release does not delete directory markers when creating
+files or directories underneath.
+This is incompatible with versions of the Hadoop S3A client released
+before 2021.
 
 Consult [Controlling the S3A Directory Marker Behavior](directory_markers.html) for
 full details.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

@@ -119,14 +119,7 @@ Without S3Guard, listing performance may be slower. However, Hadoop 3.3.0+ has s
 improved listing performance ([HADOOP-17400](https://issues.apache.org/jira/browse/HADOOP-17400)
 _Optimize S3A for maximum performance in directory listings_) so this should not be apparent.
 
-We recommend disabling [directory marker deletion](directory_markers.html) to reduce
-the number of DELETE operations made when writing files.
-this reduces the load on the S3 partition and so the risk of throttling, which can
-impact performance.
-This is very important when working with versioned S3 buckets, as the tombstone markers
-created will slow down subsequent listing operations.
-
-Finally, the S3A [auditing](auditing.html) feature adds information to the S3 server logs
+The S3A [auditing](auditing.html) feature adds information to the S3 server logs
 about which jobs, users and filesystem operations have been making S3 requests.
 This auditing information can be used to identify opportunities to reduce load.
 
@@ -162,7 +155,6 @@ Example
 ```bash
 > hadoop s3guard bucket-info -magic -markers keep s3a://test-london/
 
-2021-11-22 15:21:00,289 [main] INFO  impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be kept
 Filesystem s3a://test-london
 Location: eu-west-2
 

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md

@@ -339,16 +339,19 @@ Hadoop supports [different policies for directory marker retention](directory_ma
 -essentially the classic "delete" and the higher-performance "keep" options; "authoritative"
 is just "keep" restricted to a part of the bucket.
 
-Example: test with `markers=delete`
+
+Example: test with `markers=keep`
 
 ```
-mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete
+mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep
 ```
 
-Example: test with `markers=keep`
+This is the default and does not need to be explicitly set.
+
+Example: test with `markers=delete`
 
 ```
-mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep
+mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete
 ```
 
 Example: test with `markers=authoritative`
@@ -1268,33 +1271,6 @@ bin/hdfs fetchdt -print secrets.bin
 # expect warning "No TokenRenewer defined for token kind S3ADelegationToken/Session"
 bin/hdfs fetchdt -renew secrets.bin
 
-# ---------------------------------------------------
-# Directory markers
-# ---------------------------------------------------
-
-# require success
-bin/hadoop s3guard bucket-info -markers aware $BUCKET
-# expect failure unless bucket policy is keep
-bin/hadoop s3guard bucket-info -markers keep $BUCKET/path
-
-# you may need to set this on a per-bucket basis if you have already been
-# playing with options
-bin/hadoop s3guard -D fs.s3a.directory.marker.retention=keep bucket-info -markers keep $BUCKET/path
-bin/hadoop s3guard -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep bucket-info -markers keep $BUCKET/path
-
-# expect to see "Directory markers will be kept" messages and status code of "46"
-bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -mkdir $BUCKET/p1
-bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -mkdir $BUCKET/p1/p2
-bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -touchz $BUCKET/p1/p2/file
-
-# expect failure as markers will be found for /p1/ and /p1/p2/
-bin/hadoop s3guard markers -audit -verbose $BUCKET
-
-# clean will remove markers
-bin/hadoop s3guard markers -clean -verbose $BUCKET
-
-# expect success and exit code of 0
-bin/hadoop s3guard markers -audit -verbose $BUCKET
 
 # ---------------------------------------------------
 # Copy to from local