HADOOP-16465 listLocatedStatus() optimisation

[mukund-thakur]

Optimize S3AFileSystem.listLocatedStatus() to perform list
operations directly and then fallback to head checks for file

Ran test in ap-south-1 bucket with command:
mvn clean verify -Ds3guard -Ddynamo -Dparallel-tests -DtestsThreadCount=8

There are two failures. I am looking at them.

[mukund-thakur]

2 of my new tests are failing on command line but succeeding in IDE.

[ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnEmptyDir:141->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=0 current=1 diff=1: object_list_requests expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnNonEmptyDir:159->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=2 current=3 diff=1: object_list_requests expected:<0> but was:<1>

[mukund-thakur]

@steveloughran @bgaborg @mehakmeet . Not sure why github is not letting me add anybody as reviewer.

[steveloughran]

1. those failures happening when guarded or unguarded?
2. check what options you are passing down to the test runner in the IDE; it may be guarded or unguarded differently from the command line.

in HADOOP-13208 #1861 I'm parameterising these tests so that they will always test guarded +unguarded and dir marker keep vs delete. Makes for more complex assertions so I'm also improving how we assert metric diffs and report their failures. It'll make the suite very different, but it's the only way to have consistent estimates of the different codepath costs

[mukund-thakur]

Failures happening for guarded bucket.
This is a parameterised test which runs for both raw and guarded FS. If the guard settings are not enabled properly then tests actually skip rather than failing. So, I am not sure what am I missing here :(

[mukund-thakur]

Found the issue. Described here.
https://issues.apache.org/jira/browse/HADOOP-16979

[steveloughran]

checkstyle is unfixable and only a line length problem

./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4319:  private RemoteIterator<S3ALocatedFileStatus> getLocatedFileStatusIteratorForDir(: Line is longer than 80 characters (found 82). [LineLength]

[steveloughran]

LGTM. +1 from me, merging after a local test run

[mukund-thakur]

Thanks :)