AIP 67 - Mult Team: Updating Executors to Support Multi Team #60912
Description
Context
Each executor needs changes before it can be used in a multi-team setup. The two major requirements are:
- Ability to read team-based configuration - e.g., two CeleryExecutors may need different brokers, backends, or autoscaling settings, etc for different teams.
- The executors must be safe to run concurrently within the same scheduler process - e.g. no shared memory/queues, global DB tables, or shared filesystem, etc. This implies different things for different executors; some may need little to no changes some may need many.
This is a meta tracking Issue for which executors have been converted. Please comment if you'd like to take on a particular executor.
Task list:
Community Supported Executors:
- LocalExecutor - @o-nikolas - (AIP 67 - Multi-Team: Update LocalExecutor to support running in a multi-team config #59021)
- CeleryExecutor - @o-nikolas - (AIP-67 - Multi Team: Update Celery Executor to support multi team #60675 )
- KubernetesExecutor - @vbottu
- EdgeExecutor - @wjddn279
Third Party Executors (Providers)
Amazon
- AwsEcsExecutor - @o-nikolas - (AIP-67 - Multi-team: Per team executor config (env var only) #55003)
- AwsBatchExecutor - @anishgirianish (AIP-67 - Multi-team: AwsBatchExecutor per team executor config #60920)
- AwsLambdaExecutor - @henry3260
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.