Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values #3221

Merged
merged 4 commits into from
Jul 28, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -800,24 +800,24 @@ The following configs are related to read and write operations.

`fs.azure.io.retry.max.retries`: Sets the number of retries for IO operations.
Currently this is used only for the server call retry logic. Used within
AbfsClient class as part of the ExponentialRetryPolicy. The value should be
`AbfsClient` class as part of the ExponentialRetryPolicy. The value should be
greater than or equal to 0.

`fs.azure.io.retry.min.backoff.interval`: Sets the minimum backoff interval for
retries of IO operations. Currently this is used only for the server call retry
logic. Used within AbfsClient class as part of the EponentialRetryPolicy. This
logic. Used within `AbfsClient` class as part of the ExponentialRetryPolicy. This
value indicates the smallest interval (in milliseconds) to wait before retrying
an IO operation. The default value is 3000 (3 seconds).

`fs.azure.io.retry.max.backoff.interval`: Sets the maximum backoff interval for
retries of IO operations. Currently this is used only for the server call retry
logic. Used within AbfsClient class as part of the EponentialRetryPolicy. This
logic. Used within `AbfsClient` class as part of the ExponentialRetryPolicy. This
value indicates the largest interval (in milliseconds) to wait before retrying
an IO operation. The default value is 30000 (30 seconds).

`fs.azure.io.retry.backoff.interval`: Sets the default backoff interval for
retries of IO operations. Currently this is used only for the server call retry
logic. Used within AbfsClient class as part of the EponentialRetryPolicy. This
logic. Used within `AbfsClient` class as part of the ExponentialRetryPolicy. This
value is used to compute a random delta between 80% and 120% of the specified
value. This random delta is then multiplied by an exponent of the current IO
retry number (i.e., the default is multiplied by `2^(retryNum - 1)`) and then
Expand Down Expand Up @@ -881,7 +881,7 @@ when there are too many writes from the same process.

### <a name="securityconfigoptions"></a> Security Options
`fs.azure.always.use.https`: Enforces to use HTTPS instead of HTTP when the flag
is made true. Irrespective of the flag, AbfsClient will use HTTPS if the secure
is made true. Irrespective of the flag, `AbfsClient` will use HTTPS if the secure
scheme (ABFSS) is used or OAuth is used for authentication. By default this will
be set to true.

Expand Down
4 changes: 2 additions & 2 deletions hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -448,7 +448,7 @@ use requires the presence of secret credentials, where tests may be slow,
and where finding out why something failed from nothing but the test output
is critical.

#### Subclasses Existing Shared Base Blasses
#### Subclasses Existing Shared Base Classes
brianloss marked this conversation as resolved.
Show resolved Hide resolved

There are a set of base classes which should be extended for Azure tests and
integration tests.
Expand Down Expand Up @@ -602,7 +602,7 @@ various test combinations, it will:
2. Run tests for all combinations
3. Summarize results across all the test combination runs.

As a pre-requiste step, fill config values for test accounts and credentials
As a pre-requisite step, fill config values for test accounts and credentials
needed for authentication in `src/test/resources/azure-auth-keys.xml.template`
and rename as `src/test/resources/azure-auth-keys.xml`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,10 @@ public void testAbfsConfigConstructor() throws Exception {
ExponentialRetryPolicy policy = new ExponentialRetryPolicy(
new AbfsConfiguration(config, "dummyAccountName"));

Assert.assertEquals(expectedMaxRetries, policy.getRetryCount());
Assert.assertEquals(expectedMinBackoff, policy.getMinBackoff());
Assert.assertEquals(expectedMaxBackoff, policy.getMaxBackoff());
Assert.assertEquals(expectedDeltaBackoff, policy.getDeltaBackoff());
Assert.assertEquals("Max retry count was not set as expected.", expectedMaxRetries, policy.getRetryCount());
Assert.assertEquals("Min backoff interval was not set as expected.", expectedMinBackoff, policy.getMinBackoff());
Assert.assertEquals("Max backoff interval was not set as expected.", expectedMaxBackoff, policy.getMaxBackoff());
Assert.assertEquals("Delta backoff interval was not set as expected.", expectedDeltaBackoff, policy.getDeltaBackoff());
}

private AbfsConfiguration getAbfsConfig() throws Exception {
Expand Down