[AIRFLOW-5889] Fix polling for AWS Batch job status #6765
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dazza-codes
force-pushed
the
aws-batch-poll-exceptions
branch
2 times, most recently
from
7c29b4d to
7f65c55
Compare
Codecov Report
@@ Coverage Diff @@
## master #6765 +/- ##
=========================================
- Coverage 84.32% 84.3% -0.02%
=========================================
Files 672 672
Lines 38179 38210 +31
=========================================
+ Hits 32195 32214 +19
- Misses 5984 5996 +12
Continue to review full report at Codecov.
|
Contributor
Author
|
The bug fix in this PR should get into a 1.x release, so it might be better if this PR sneaks in before the AIP-21 stuff in #6764 is resolved. |
dazza-codes
changed the title
dazza-codes
changed the title
ashb
reviewed
Dec 11, 2019
ashb
reviewed
Dec 11, 2019
Member
ashb
left a comment
ashb
left a comment
There was a problem hiding this comment.
LGTM - one small change requested, then we can merge this.
dazza-codes
changed the title
dimberman
previously requested changes
Dec 11, 2019
- errors in polling for job status should not fail
the airflow task when the polling hits an API throttle
limit; polling should detect those cases and retry a
few times to get the job status, only failing the task
when the job description cannot be retrieved
- added typing for the BatchProtocol method return
types, based on the botocore.client.Batch types
- applied trivial format consistency using black, i.e.
$ black -t py36 -l 96 {files}
dazza-codes
force-pushed
the
aws-batch-poll-exceptions
branch
from
7f65c55 to
c52463e
Compare
dazza-codes
changed the title
5 tasks
kaxil
pushed a commit
that referenced
this pull request
Dec 17, 2019
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
5 tasks
kaxil
pushed a commit
that referenced
this pull request
Dec 18, 2019
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
ashb
pushed a commit
that referenced
this pull request
Dec 19, 2019
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
galuszkak
pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
…pache#6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR target
This PR targets a 1.10.x release for this bug fix
1.10.testbranch instead ofmaster?Jira
Possible conflicts
Description
the airflow task when the polling hits an API throttle
limit; polling should detect those cases and retry a
few times to get the job status
types, based on the botocore.client.Batch types
$ black -t py36 -l 96 {files}
Tests
Commits
Documentation