HADOOP-16499. S3A retry policy to be exponential #1246
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-exponential retry policy used -retry interval = 1s -retry attempts = 8 -docs updated Change-Id: I48b17af34a50b7291f2c360ee74dbce7540556b4
steveloughran
added
enhancement
fs/s3
…d were generating failures Change-Id: Id3692c29584d08749e1a78f4c95c3298a61672b7
|
test run has shown some tuning is needed on those which simulate failures |
Change-Id: Idb034b81c12246bcb69de050ac91643c6ac34327
|
latest run against ireland with s3guard and versioning |
Change-Id: I2c62aceb344a7bb531860651babce9ff85486df5
|
For reviewers: the reason I move off the |
|
two other tests timed out on a test run |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Tuning some tests to cope with shorter timeouts by setting the timeouts to ranges which they are happy with. For ITestS3AInconsistency that means we want S3Guard to time out eventually. All the various values are no all configured in one place so their requirements are visible. Change-Id: I8fffc95d79a74f371b2ff78271e58530857b73cd
|
🎊 +1 overall
This message was automatically generated. |
|
Turns out there are even more tests which have expectations that connectivity/read exceptions retry for a time less than the test timeout, i.e. the default of 20s. Shrink the retry time to a couple of seconds and the tests are faster |
it turns out that with a long retry interval, various tests which were expecting failures, be it in network setup or OOB tests were using the default retry times -extend them and tests take longer, sometimes so much that they time out. This patch cranks back the retry intervals on more tests to fail much faster than before, which is actually an improvement. Considered but not done: changing the defaults in the AbstractS3ATestBase; It seems better to use the real defaults and identify those tests which expect to retry a lot Change-Id: I1e9bac752907a6ced8a0bc7c37c95006dd8d6e98
|
final test run: all good. test failure in [ERROR] {{test_300_MetastorePrune}} which is HADOOP-16481 with pr #1209 |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
one question for reviewers, what are the good initial values? with 500ms and 7 attempts, that's 64s of delay, with 32s being the final pause. I don't want to massively expand the time to fail beyond 1m as this timeout also handles things like time to give up on network errors and we don't want to block for vast amounts of time there. |
|
Change seems good to me +1 on it.
looks reasonable. I'm running the test now, I'll be back with the results soon. |
|
thx. i will wait for your test run before committing |
|
Running tests against ireland, I got the following error: Otherwise nothing else. This failure is intermittent and I will not show on re-run. |
|
seems good to me, +1; will commit soon. |
amahussein
pushed a commit
to amahussein/hadoop
that referenced
this pull request
Oct 29, 2019
…ibuted by Steve Loughran.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
-exponential retry policy used
-retry interval = 1s
-retry attempts = 8
-docs updated
tested s3 ireland
Change-Id: I48b17af34a50b7291f2c360ee74dbce7540556b4