HADOOP 17290 to 17811. ABFS Backports

[sumangala-patki]

PR contains cherry-pick of the following ABFS driver commits:

• HADOOP-17290. ABFS: Add Identifiers to Client Request Header #2520
• HADOOP-17765. ABFS: Use Unique File Paths in Tests #3153
• HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values #3221

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 44 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	37m 20s		branch-3.3 passed
+1 💚	compile	0m 33s		branch-3.3 passed
+1 💚	checkstyle	0m 25s		branch-3.3 passed
+1 💚	mvnsite	0m 39s		branch-3.3 passed
+1 💚	javadoc	0m 29s		branch-3.3 passed
+1 💚	spotbugs	1m 4s		branch-3.3 passed
+1 💚	shadedclient	19m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	compile	0m 33s		the patch passed
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed
+1 💚	spotbugs	1m 20s		the patch passed
+1 💚	shadedclient	20m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 14s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		101m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3253/1/artifact/out/Dockerfile
GITHUB PR	#3253
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux e96fdeec2246 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 1ff850a
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3253/1/testReport/
Max. process+thread count	549 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3253/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[sumangala-patki]

TEST RESULTS:

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

AppendBlob-HNS-OAuth

[ERROR] Failures: 
[ERROR]   ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166 » IO
[ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut 
[ERROR] Tests run: 271, Failures: 0, Errors: 1, Skipped: 76

HNS-OAuth

[ERROR] Failures: 
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40].
[ERROR] Tests run: 103, Failures: 1, Errors: 0, Skipped: 1
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 98
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 271, Failures: 0, Errors: 1, Skipped: 52

HNS-SharedKey

[WARNING] Tests run: 103, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Failures: 
[ERROR]   ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:93->testWithDifferentCustomTokenFetchRetry:122->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0)
[ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 54
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 271, Failures: 0, Errors: 2, Skipped: 40

NonHNS-SharedKey

[ERROR] Failures: 
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 14 is not within the expected range: [5.60, 8.40].
[ERROR] Tests run: 103, Failures: 1, Errors: 0, Skipped: 2
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 276 
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 271, Failures: 0, Errors: 2, Skipped: 40

[steveloughran]

I've already merged the HADOOP-17811 retry one as it was painless; doing a distcp one next. The big troublespot is the paths one...can you provide that as a PR all on its own? thanks

[sumangala-patki]

I've already merged the HADOOP-17811 retry one as it was painless; doing a distcp one next. The big troublespot is the paths one...can you provide that as a PR all on its own? thanks

Sure! The paths one conflicts with the missing Correlation ID PR in 3.3, so that one (#2520) needs to go in first. Will raise backport PRs for both