Skip to content

AIP-67 - Multi Team: Update AwsLambdaExecutor to support multi team#61321

Merged
o-nikolas merged 1 commit intoapache:mainfrom
henry3260:multi-team-aws-lambda-executor
Feb 19, 2026
Merged

AIP-67 - Multi Team: Update AwsLambdaExecutor to support multi team#61321
o-nikolas merged 1 commit intoapache:mainfrom
henry3260:multi-team-aws-lambda-executor

Conversation

@henry3260
Copy link
Contributor

@henry3260 henry3260 commented Feb 1, 2026

Changes

Add supports_multi_team: bool = True flag to AwsLambdaExecutor

Update all conf.get() calls to use self.conf.get() for team-specific configuration

Add test_team_config test to verify team isolation works

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look mostly good to me, have you done any manual testing? The most important piece is back compat testing with 2.11 airflow.

@henry3260 henry3260 force-pushed the multi-team-aws-lambda-executor branch from aa136f7 to 5ce81c4 Compare February 7, 2026 13:32
Enable AwsLambdaExecutor to read team-based config correctly
@henry3260 henry3260 force-pushed the multi-team-aws-lambda-executor branch from 5ce81c4 to 2571e25 Compare February 7, 2026 14:21
@henry3260
Copy link
Contributor Author

henry3260 commented Feb 7, 2026

The changes look mostly good to me, have you done any manual testing? The most important piece is back compat testing with 2.11 airflow.

Yeah, I validated the multi-team configuration precedence in an Airflow 2.11 environment using a small smoke-style check (I set team and global env vars inside a 2.11 container and inspected the resolved values). This verifies that team-specific keys override global keys while unspecified keys correctly fall back to global.

@o-nikolas
Copy link
Contributor

The changes look mostly good to me, have you done any manual testing? The most important piece is back compat testing with 2.11 airflow.

Yeah, I validated the multi-team configuration precedence in an Airflow 2.11 environment using a small smoke-style check (I set team and global env vars inside a 2.11 container and inspected the resolved values). This verifies that team-specific keys override global keys while unspecified keys correctly fall back to global.

Thanks for doing that @henry3260! Did you actually run any dags with the executor to ensure task execution is working end to end?

@henry3260
Copy link
Contributor Author

The changes look mostly good to me, have you done any manual testing? The most important piece is back compat testing with 2.11 airflow.

Yeah, I validated the multi-team configuration precedence in an Airflow 2.11 environment using a small smoke-style check (I set team and global env vars inside a 2.11 container and inspected the resolved values). This verifies that team-specific keys override global keys while unspecified keys correctly fall back to global.

Thanks for doing that @henry3260! Did you actually run any dags with the executor to ensure task execution is working end to end?

Not yet. Since I'm still getting familiar with the AWS setup, I'll need a bit more time to run the end-to-end tests. I'll update you as soon as I have the results.

@o-nikolas
Copy link
Contributor

The changes look mostly good to me, have you done any manual testing? The most important piece is back compat testing with 2.11 airflow.

Yeah, I validated the multi-team configuration precedence in an Airflow 2.11 environment using a small smoke-style check (I set team and global env vars inside a 2.11 container and inspected the resolved values). This verifies that team-specific keys override global keys while unspecified keys correctly fall back to global.

Thanks for doing that @henry3260! Did you actually run any dags with the executor to ensure task execution is working end to end?

Not yet. Since I'm still getting familiar with the AWS setup, I'll need a bit more time to run the end-to-end tests. I'll update you as soon as I have the results.

Thanks, much appreciated @henry3260

@o-nikolas o-nikolas added the full tests needed We need to run full set of tests for this PR to merge label Feb 18, 2026
@henry3260
Copy link
Contributor Author

henry3260 commented Feb 19, 2026

image image image

Hi @o-nikolas , I think it works fine in 2.11!

@henry3260
Copy link
Contributor Author

airflow % docker exec airflow-scheduler airflow dags list-runs -d test_lambda_executor
/home/airflow/.local/lib/python3.12/site-packages/airflow/metrics/base_stats_logger.py:22 RemovedInAirflow3Warning: Timer and timing metrics publish in seconds were deprecated. It is enabled by default from Airflow 3 onwards. Enable timer_unit_consistency to publish all the timer and timing metrics in milliseconds.
/home/airflow/.local/lib/python3.12/site-packages/airflow/triggers/base.py:27 RemovedInAirflow3Warning: Timer and timing metrics publish in seconds were deprecated. It is enabled by default from Airflow 3 onwards. Enable timer_unit_consistency to publish all the timer and timing metrics in milliseconds.
dag_id               | run_id                            | state   | execution_date            | start_date                       | end_date                        
=====================+===================================+=========+===========================+==================================+=================================
test_lambda_executor | manual__2026-02-19T17:30:47+00:00 | success | 2026-02-19T17:30:47+00:00 | 2026-02-19T17:31:01.086388+00:00 | 2026-02-19T17:31:23.804851+00:00
test_lambda_executor | manual__2026-02-19T17:30:33+00:00 | success | 2026-02-19T17:30:33+00:00 | 2026-02-19T17:30:44.678423+00:00 | 2026-02-19T17:31:06.547126+00:00
                                                                                                                                                                    

@o-nikolas
Copy link
Contributor

Hi @o-nikolas , I think it works fine in 2.11!

Lovely! Thanks for doing the back compat testing 😄

@o-nikolas o-nikolas merged commit 2935876 into apache:main Feb 19, 2026
90 checks passed
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
Enable AwsLambdaExecutor to read team-based config correctly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers full tests needed We need to run full set of tests for this PR to merge provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants