AIP-67 - Multi Team: Update Celery Executor to support multi team

[o-nikolas]

Updating the Celery executor to read from team based config and also support multiple instances running concurrently.

The latter is the largest source of changes. Much of the celery configuration (both Airflow config and Celery config) was module based. Modules are cached and shared in Python. So the majority of the changes are moving that module level config code to be function based (while trying to also maintain backwards compatibility).

The way Celery tasks are sent to workers also changed as a consequence of this. Since sending tasks is parallelized with multiple processes (which do not share memory with the parent) the send task logic now re-creates a celery app in the sub processes (since the pickling and unpickling that python does to try pass state to the sub processes was not reliably creating the correct celery app objects).

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  Cline

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

Thanks for the enhancements! the current implementation introduces static type errors that will fail our CI pipeline. I have provided inline patches to correct the type signatures and ensure compliance with our MyPy configuration.

[dheerajturaga]

@o-nikolas I have tested the general functionality of celery worker with your changes as well as tested out the CLI. Things are working as expected. However, I don't see how I can test the --team option. Seems like if I set a team the jobs still seem to be launching on the worker. How do you propose I test the multi team support here?

[o-nikolas]

@o-nikolas I have tested the general functionality of celery worker with your changes as well as tested out the CLI. Things are working as expected. However, I don't see how I can test the --team option. Seems like if I set a team the jobs still seem to be launching on the worker. How do you propose I test the multi team support here?

Thanks for the thorough review @dheerajturaga! I've addressed those issues (slightly differently for one than your suggested patch). I'm currently struggling with back compat tests. It's slow/difficult because those tests to do not run successfully in breeze on my laptop. So I have to push to the PR to test each change.

As far as testing with the --team flag. For this you have to have a full multi-team setup, which we don't have great documentation for yet (coming soon). The most helpful testing is actually on the backcompat side (testing with airflow 2.11 and 3.1.X)

[dheerajturaga]

@o-nikolas , I have run backward compatibility checks. Things work good in 2.11.0 however when I tried this with 3.1.3 I found issues. There are incomplete API contract in the ExecutorConf class between Airflow versions.

[dheerajturaga]

Hopefully the final set of changes needed to be consistent. Everything else looks good.

Request for changes has been left after fixes were applied.

[dheerajturaga]

Awesome! Thanks so much for your patience, I know there was a lot of back and forth. Current changes look good!