Skip to content

Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs#55703

Merged
o-nikolas merged 12 commits into
apache:mainfrom
yash1thsa:fix/54561
Sep 17, 2025
Merged

Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs#55703
o-nikolas merged 12 commits into
apache:mainfrom
yash1thsa:fix/54561

Conversation

@yash1thsa
Copy link
Copy Markdown
Contributor

Feel free to ping committers for the review!

In case of an existing issue, reference it using one of the following:

closes: #54561

Issue:
AWS BatchOperator does not fetch log entries for deferred jobs. The issue is prevelant in MWAA and other deployment as well.

Steps:

  • Reproduced the issue in local with AWS connection
  • Code: For deferred path in batch operator module, add monitor_job function as a downstream step
  • Tested the fix with both test cases and running in local with AWS connection

Tests:

  • Happy path with awslog_enabled set to True
  • Happy path with awslog_enabled set to False
  • Failure step for validation error

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Sep 16, 2025
@eladkal eladkal requested a review from vincbeck September 16, 2025 02:31
@vincbeck
Copy link
Copy Markdown
Contributor

Static checks are failing, please fix them. If you're not familiar, you can read the doc about it.

@yash1thsa
Copy link
Copy Markdown
Contributor Author

Static checks are failing, please fix them. If you're not familiar, you can read the doc about it.

@vincbeck Thanks for document. Let me review.

@yash1thsa yash1thsa force-pushed the fix/54561 branch 2 times, most recently from 5bc47fb to b489407 Compare September 16, 2025 18:54
@yash1thsa
Copy link
Copy Markdown
Contributor Author

Cleaned up all CI errors and rebased to master. Please review and let me know for any issues

@vincbeck vincbeck requested a review from o-nikolas September 17, 2025 16:38
@o-nikolas o-nikolas merged commit d18e3dc into apache:main Sep 17, 2025
78 checks passed
@boring-cyborg
Copy link
Copy Markdown

boring-cyborg Bot commented Sep 17, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS BatchOperator does not fetch log entries for deferred jobs

3 participants