-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HADOOP-18647. x-ms-client-request-id to have some way that identifies retry of an API. (#5437) #5498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-18647. x-ms-client-request-id to have some way that identifies retry of an API. (#5437) #5498
Conversation
…apache#5437) The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous operation Contributed by Pranav Saxena
🎊 +1 overall
This message was automatically generated. |
:::: AGGREGATED TEST RESULT :::: HNS-OAuth[INFO] Results: HNS-SharedKey[INFO] Results: NonHNS-SharedKey[INFO] Results: AppendBlob-HNS-OAuth[INFO] Results: Time taken: 44 mins 54 secs. |
🎊 +1 overall
This message was automatically generated. |
@steveloughran , requesting you to kindly review the PR please. Thank you. |
+1, merged |
Merged PR on trunk: #5437
JIRA: https://issues.apache.org/jira/browse/HADOOP-18647
Merged commit in trunk: 759ddeb
Description:
In case primaryRequestId in x-ms-client-request-id is empty-string, the retry's primaryRequestId has to contain last part of clientRequestId UUID.
What we want to have?
1.
:a68ddb84-cde5-49a1-84c8-61a668e1fa4c:bda15af3-bef2-4696-97f5-76222fc02bbe:::CF:0
2.
:14673571-509b-4a3c-a0c5-ee0ab5c4ed59:bda15af3-bef2-4696-97f5-76222fc02bbe:61a668e1fa4c::CF:1_RT
3. Here,
a68ddb84-cde5-49a1-84c8-61a668e1fa4c
is the clientRequestId of the first try(retry =0), and it has emptyString primaryId. In the second try(retry =1), primaryId is :61a668e1fa4c
(last part of retry_0's clientRequestId UUID):d5305735-9096-4cf2-bdef-b26f1ebf8922:40c2a286-f60d-41ea-9228-79cd66730901:6c68acd9-6a37-40d5-93a0-05358fbc8203::CF:0
:de56b0c8-fe2c-4496-96eb-939d7d7d42e9:40c2a286-f60d-41ea-9228-79cd66730901:6c68acd9-6a37-40d5-93a0-05358fbc8203::CF:1_RT
6c68acd9-6a37-40d5-93a0-05358fbc8203
remains same throughout retries.More real example, explicitly made readTimeout to 10ms.:
:16f81279-803f-41f4-a17e-1478cd618f2b:b6e030dc-6842-4e6e-ad86-8448b15a4be1:::CF:0
:b93b5142-a7b3-404d-82d6-4e5a47322923:b6e030dc-6842-4e6e-ad86-8448b15a4be1:1478cd618f2b::CF:1_RT
:f8bd21cf-d379-4c56-bbd9-bd893ef95e06:b6e030dc-6842-4e6e-ad86-8448b15a4be1:1478cd618f2b::CF:2_RT
:::: AGGREGATED TEST RESULT ::::
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 11 is not within the expected range: [5.60, 8.40].
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 138, Failures: 2, Errors: 1, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 3, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[INFO]
[ERROR] Tests run: 336, Failures: 0, Errors: 1, Skipped: 55
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 11 is not within the expected range: [5.60, 8.40].
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 138, Failures: 2, Errors: 1, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 1, Skipped: 54
[INFO] Results:
[INFO]
[WARNING] Tests run: 336, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 138, Failures: 1, Errors: 1, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=92).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345 » TestTimedOut
[INFO]
[ERROR] Tests run: 568, Failures: 1, Errors: 1, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://testcontainer@pranavsaxenanonhns.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation
abfs:/... [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ... [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation
...[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 8, Skipped: 46
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 13 is not within the expected range: [5.60, 8.40].
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 138, Failures: 2, Errors: 1, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 3, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[INFO]
[ERROR] Tests run: 336, Failures: 0, Errors: 1, Skipped: 55