Skip to content

Conversation

@anishgirianish
Copy link
Contributor

@anishgirianish anishgirianish commented Jan 22, 2026


Summary

Add multi-team support to the AWS Batch Executor,

Changes

  • Add supports_multi_team: bool = True flag to AwsBatchExecutor
  • Move class-level MAX_SUBMIT_JOB_ATTEMPTS to instance-level max_submit_job_attempts
  • Update all conf.get() calls to use self.conf.get() for team-specific configuration
  • Change _load_submit_kwargs() from static method to instance method
  • Update build_submit_kwargs(conf) to take conf as required parameter
  • Add test_team_config test to verify team isolation works

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jan 22, 2026
@o-nikolas
Copy link
Contributor

Thanks for the PR, I will review it in the next day or so!

Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good. Did you do any manual testing? If so can you describe the testing you did and the results you saw?

@anishgirianish
Copy link
Contributor Author

anishgirianish commented Jan 24, 2026

Hi @o-nikolas, thanks for the review!

Yes, I did some manual testing. Set up AWS Batch (Fargate) to verify the changes work end-to-end.

Built a Docker image with the PR changes and ran it on AWS Batch.

Job succeeded:

aws-batch-console-job-succeeded

Logs output:

cloud-watch-logs-output

The logs show:
AwsBatchExecutor imported successfully
supports_multi_team: True

Image in ECR:

ecr-repository-image-upload

Everything works - the executor loads correctly with supports_multi_team enabled, and the config changes (moving to
self.conf and instance-level max_submit_job_attempts) don't break anything.

The test_team_config unit test covers the team isolation logic.

Let me know if you'd like me to test anything else!

Thank you.

Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for testing and contribution!

Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable.

@ferruzzi ferruzzi merged commit 75b4a29 into apache:main Jan 26, 2026
89 checks passed
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants