Hadoop-17015. ABFS: Handling Rename and Delete idempotency #2021

snvijaya · 2020-05-13T17:34:01Z

Requests failing due to server timeouts and network failures are retried. At times the request might have succeeded at the server end when client received a timeout. In such case, a retry done for Rename or Delete operation will end in an user error for HTTP 404 (Not Found). In case of Rename, user error is NOT_FOUND as the source file is already renamed and is no longer available.

When the user error is thrown back to the calling application it mostly leads to job failure and is hard to debug from client side on how rename or delete ended up with an user error of file/folder not being present.

Rest of the PUT/POST operations get executed again at server end and maintains the idempotency state and needs no handling in ABFS Driver.

In this PR:

Rename: If a rename request was retried and has failed with HTTP 404 status, LMT on the destination path is checked. If LMT is post the start time of the first rename request or is within a bracket of 5 mins previous (to absorb clock skew on server side), it is assumed that the earlier rename operation was successful at backend and hence success is returned.
Delete: Delete is considered idempotent by default. Hence if a delete request had to be retried and has failed with HTTP 404 status, a success response is returned.

Mock tests to mimic a retried rename/delete request are added.

snvijaya · 2020-05-13T17:35:05Z

Test results from accountin East US 2.

Test results from HNS enabled account:
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 74
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

Test results from Non-HNS enabled account:
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 248
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/DateTimeUtils.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

bilaharith · 2020-05-13T18:48:38Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

    return op;
  }

+  /**


Would it be better to move the idempotency related methods to a separate Utility/Helper class?

The changes are not utility related and are insync with the handling of the ABFS response. The reason they were included as separate methods was to enable mock testing which was otherwise not possible. Retaining the change as such.

One option is restrict the method accessibility tp package level and use @VisibleForTesting annotation. Keeping a method public solely for testing doesn't look good practice.

Also the idea is, If you have methods 'assisting', chances are the class is actually doing too much. Moving these methods into separate classes with public interfaces keeps the class with the assisting methods responsible for one thing and one thing only (see Single Responsibility Principle). This move into separte classes automatically makes for a testable structure as your methods must be made public

https://softwareengineering.stackexchange.com/questions/274937/is-it-bad-practice-to-make-methods-public-solely-for-the-sake-of-unit-testing

AbfsClient class handles triggering of requests to Store backend and returning the AbfsRestOperation back. For Rename and Delete, the response to return is not determined if request was re-tried by the idempotent logic. It will be not right to consider these methods as "assisting" or providing a utility service and are part of the actual flow.

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

hadoop-yetus · 2020-05-13T19:03:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	markdownlint	0m 1s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 1s	trunk passed
+1 💚	compile	0m 33s	trunk passed
+1 💚	checkstyle	0m 24s	trunk passed
+1 💚	mvnsite	0m 36s	trunk passed
+1 💚	shadedclient	14m 55s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 28s	trunk passed
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 49s	trunk passed
-0 ⚠️	patch	1m 9s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 16s	the patch passed
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 58s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 16s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		57m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/1/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux 1967b584bf0d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `108ecf9`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/1/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bilaharith · 2020-05-13T19:14:35Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

+    testRenameTimeout(HTTP_NOT_FOUND, HTTP_NOT_FOUND, true);
+  }
+
+  private void testRenameTimeout(


Couldn't find it reasonable to keep all the cases into the same method with if-else. I think separating the same could improve readability of the test cases.

Over the past PRs we have had several instances of reusable code being duplicated. These test cases share a lot of reusable code, hence the reusable code is put into a method. Have made some minor updates which might help improve readability.

I see common code as (1. Creating FS instance, 2. Creating testClient instance, 3. Creating mock AbfsRestOperation instance, 4. Common assertion). This is clubbed with the non common codes (1. Creating http400Op instance for one particular test case, 2. Creating http404Op instance for another particular test case)
Shall we move the common instances that are required to a separate private methods and call those methods in each test case? That would help improve readability and solved the problem of code duplication.

I think we should relook and address the issues with the old PRs if those are recent, where resusable code is duplicated. Could you please create a workitem for this.

Creating a method, passing over all indiividual mock instances created back to test method will actually render code more unreadable. Will leave this comment open if any other reviewer feels the code is unreadable too.

For issues that have come across for my code changes, i have made modifications with the respective PRs. Should definitely keep a look out for any that come across.

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

bilaharith

Please check the comments

bilaharith

Change the heading as follows.
HADOOP-17015. ABFS: Handling Rename and Delete idempotency

bilaharith · 2020-05-13T19:25:04Z

Requests failing due to server timeouts and network failures are exponentially retries. At times the request might have succeeded at the server end when client received a timeout. In such case, a retry done for Rename or Delete operation will end in an user error for HTTP 404 (Not Found). In case of Rename, user error is NOT_FOUND as the source file is already renamed and is no longer available.

The following doesn't look right.
"Requests failing due to server timeouts and network failures are exponentially retries."

snvijaya · 2020-05-14T08:31:31Z

@bilaharith Comments have been addressed. Please take a look.

For the latest

Requests failing due to server timeouts and network failures are exponentially retries. At times the request might have succeeded at the server end when client received a timeout. In such case, a retry done for Rename or Delete operation will end in an user error for HTTP 404 (Not Found). In case of Rename, user error is NOT_FOUND as the source file is already renamed and is no longer available.

The following doesn't look right.
"Requests failing due to server timeouts and network failures are exponentially retries."

Was a typo for 'retried'. Fixed it.

snvijaya · 2020-05-14T08:31:57Z

Change the heading as follows.
HADOOP-17015. ABFS: Handling Rename and Delete idempotency

I believe you were highlighting a space needed before ABFS. Have made that change.

snvijaya · 2020-05-14T08:32:48Z

Test results from latest updates:
HNS
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 74
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

Non-HNS
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 248
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

hadoop-yetus · 2020-05-14T09:40:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 54s	trunk passed
+1 💚	compile	0m 27s	trunk passed
+1 💚	checkstyle	0m 21s	trunk passed
+1 💚	mvnsite	0m 30s	trunk passed
+1 💚	shadedclient	16m 19s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	trunk passed
+0 🆗	spotbugs	0m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 49s	trunk passed
-0 ⚠️	patch	1m 6s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 29s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 21s	the patch passed
+1 💚	findbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 20s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		63m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/2/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux 07f8beb828a1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `0918433`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/2/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

bilaharith

Please see the comments

bilaharith

Please check the comments

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java

bilaharith

Please check the comment

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java

hadoop-yetus · 2020-05-14T18:39:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
-1 ❌	mvninstall	19m 30s	root in trunk failed.
+1 💚	compile	0m 33s	trunk passed
+1 💚	checkstyle	0m 26s	trunk passed
+1 💚	mvnsite	0m 36s	trunk passed
-1 ❌	shadedclient	15m 44s	branch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	0m 53s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 52s	trunk passed
-0 ⚠️	patch	1m 10s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 17s	the patch passed
+1 💚	mvnsite	0m 28s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	shadedclient	14m 17s	patch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 16s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		59m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/3/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux 56f85a69f859 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `017d24e`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/3/artifact/out/branch-mvninstall-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/3/testReport/
Max. process+thread count	413 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-azure/src/site/markdown/abfs.md

bilaharith

Please check the comments

snvijaya

Please find responses above.

For yetus run, the failure is for shaded client.
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 2.195 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [02:14 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:18 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... FAILURE [ 1.539 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [04:03 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . FAILURE [ 0.200 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 22.661 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:14 min
[INFO] Finished at: 2020-05-14T18:18:19Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (check-jar-contents) on project hadoop-client-check-invariants: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]

Failure is for enforce-banned-dependecies on hadoop-client-check-invariants. There are no updates to dependencies in this PR and even if hadoop-azure is a dependent module for hadoop-client-invariants.

Please go ahead and review and I will sort this one out of one of the committers. (to see if any action is needed.)

bilaharith · 2020-05-15T08:09:39Z

...azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java

@@ -182,4 +182,11 @@ public void testSSLSocketFactoryConfiguration()
    assertEquals(DelegatingSSLSocketFactory.SSLChannelMode.OpenSSL, localAbfsConfiguration.getPreferredSSLFactoryOption());
  }

+  public static AbfsConfiguration updateRetryConfigs(AbfsConfiguration abfsConfig,


Not sure if this class is the right place for this method.

This is the unit test class for AbfsConfiguration. I do not want to create separate utility class just to return a test instance of main class.

bilaharith · 2020-05-15T08:13:03Z

...-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java

@@ -240,4 +242,25 @@ public void verifyUserAgentClusterType() throws Exception {
      .contains(DEFAULT_VALUE_UNKNOWN);
  }

+  public static AbfsClient createTestClientFromCurrentContext(


Same doubt as the above comment.
TestAbfsClient should have test methods the AbfsClient and supporting methods.
This method looks more like a Utility.

Again, dont want to create a separate test utility class for AbfsClient alone to return a test instance and hence have placed it in the unit test class for AbfsClient.
For future tests that will need to mock or create new instances, it will be easy to check respective unit test class for any method available.

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java

bilaharith

LGTM

snvijaya · 2020-05-15T08:56:49Z

Latest test results:
HNS
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 74
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

Non-HNS
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 436, Failures: 0, Errors: 0, Skipped: 248
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

hadoop-yetus · 2020-05-15T09:05:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	24m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
-1 ❌	mvninstall	20m 6s	root in trunk failed.
+1 💚	compile	0m 31s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 34s	trunk passed
-1 ❌	shadedclient	15m 20s	branch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 28s	trunk passed
+0 🆗	spotbugs	0m 53s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 50s	trunk passed
-0 ⚠️	patch	1m 9s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 16s	the patch passed
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	shadedclient	15m 38s	patch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	1m 1s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 18s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 32s	The patch does not generate ASF License warnings.
		84m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/4/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux 264ad583d743 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ac4a2e1`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/4/artifact/out/branch-mvninstall-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/4/testReport/
Max. process+thread count	413 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-05-15T10:10:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
-1 ❌	mvninstall	19m 49s	root in trunk failed.
+1 💚	compile	0m 31s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 36s	trunk passed
-1 ❌	shadedclient	15m 28s	branch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 29s	trunk passed
+0 🆗	spotbugs	0m 53s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 51s	trunk passed
-0 ⚠️	patch	1m 9s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 16s	the patch passed
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	shadedclient	14m 16s	patch has errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	0m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 16s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		59m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/5/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux 87ccc5cca2e9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ac4a2e1`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/5/artifact/out/branch-mvninstall-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/5/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/5/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

DadanielZ · 2020-05-19T03:58:16Z

Looks good! @snvijaya please resolve the conflicts then I will help to commit.

snvijaya · 2020-05-19T15:32:56Z

Thanks @DadanielZ . Have resolved conflicts and updated PR.

hadoop-yetus · 2020-05-19T16:20:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	28m 24s	Docker failed to build yetus/hadoop:81d8b715346.

Subsystem	Report/Notes
GITHUB PR	#2021
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/6/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-05-19T19:26:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 53s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	markdownlint	0m 1s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 54s	trunk passed
+1 💚	compile	0m 28s	trunk passed
+1 💚	checkstyle	0m 21s	trunk passed
+1 💚	mvnsite	0m 30s	trunk passed
+1 💚	shadedclient	16m 4s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	trunk passed
+0 🆗	spotbugs	0m 53s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 51s	trunk passed
-0 ⚠️	patch	1m 7s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 22s	the patch passed
+1 💚	javac	0m 22s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 58s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 22s	the patch passed
+1 💚	findbugs	1m 1s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 20s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		66m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/7/artifact/out/Dockerfile
GITHUB PR	#2021
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname	Linux aa3589dcdf5c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d4e3640`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/7/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2021/7/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

DadanielZ · 2020-05-19T19:29:28Z

LGTM, +1. Thanks!

snvijaya added 5 commits April 29, 2020 22:05

rename changes

5a11d4d

Some refactoring

6471661

Merge from master

16f0b7b

Checkstyle and findbugs fixes

c158597

Removing redundant new lines

77f54cb