Missing CeleryKubernetesExecutor

[GrumpyCat51]

Official Helm Chart version

1.16.0 (latest released)

Apache Airflow version

3.0.0

Kubernetes Version

Minikube v1.35.0

Helm Chart configuration

executor: "CeleryKubernetesExecutor"

Docker Image customizations

No response

What happened

When selecting the CeleryKubernetesOperator as specified in the comment just above the executor setting, both the scheduler as well as the celery worker don't start as the executor cannot be loaded properly:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/module_loading.py", line 35, in import_string
    module_path, class_name = dotted_path.rsplit(".", 1)
    ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 55, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 48, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 111, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler
    run_command_with_daemon_option(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
    callback()
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda>
    callback=lambda: _run_scheduler_job(args),
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job
    with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/scheduler_command.py", line 65, in _serve_logs
    executor_class, _ = ExecutorLoader.import_default_executor_cls()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py", line 299, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py", line 289, in import_executor_cls
    return import_string(executor_name.module_path), executor_name.connector_source
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/module_loading.py", line 37, in import_string
    raise ImportError(f"{dotted_path} doesn't look like a module path")
ImportError: CeleryKubernetesExecutor doesn't look like a module path

In version 2.10.5, the mapping of executors was set here:
https://github.com/apache/airflow/blob/2.10.5/airflow/executors/executor_loader.py#L63-L74
In version 3.0.0, this is severly reduced to
https://github.com/apache/airflow/blob/3.0.0/airflow-core/src/airflow/executors/executor_loader.py#L56-L61

I'd be happy to create a PR for this, but I'm not quite sure how it was intended to look like. Should the CeleryKubernetesExecutor be dropped in favor of specifying both executors individually, are they simply missing from in version 3.0.0, or should the logic figure out that CeleryKubernetesExecutor should split into CeleryExecutor and KubernetesExecutor?

What you think should happen instead

No response

How to reproduce

Run the helm chart with executor: "CeleryKubernetesExecutor"

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[kaxil]

Pass the full path to CeleryKubernetesExecutor: airflow.providers.celery.executors.celery_kubernetes_executor.CeleryKubernetesExecutor and that should fix it for now until we investigate

[kaxil]

Yeah, looks like it was an intentional change in #47322 -- we should change the comment in Helm Chart accordingly.

cc @o-nikolas

[GrumpyCat51]

I can start working on a corresponding pull request, but the helm chart should be compatible with both v2.x and v3.0, right?
The CeleryKubernetesOperator is still mentioned in quite some places throughout the documentation, so I'd assume they should all be changed.

[kaxil]

Yeah it is, the path behaviour should work with both 2.x and 3.x. The Helm chart also doesn't need to change from code POV -- just the comment that you linked

I can start working on a corresponding pull request, but the helm chart should be compatible with both v2.x and v3.0, right?

[o-nikolas]

Hey folks, checking in (I've been sick for the past few days).

The old static/hardcoded hybrid executors are deprecated and should not be used any longer. Multi Executor Config should be used instead (docs, dev email discussion thread). Even if you provide the module path to the deprecated CeleryKubernetesExecutor, it will not function on Airtflow >=3.0.0 since the hardcoded hacks to support it were removed.

It will still function on Airflow 2 of course, though I'm not familiar with helm charts and how support for both AF2 and AF3 works.

[GrumpyCat51]

Hello @o-nikolas
Would it be worth adding a sentence about the CeleryKubernetesExecutor to the update guide https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html#breaking-changes? I just image trying to upgrade the instances I've managed at my previous job that used the CeleryKubernetesExecutor, and I don't think going though the upgrade guide I would have figured out what to do or where to look. I very much like the change, but a pointer to the doc would be appreciated in my view (or a check in the helm templates like https://github.com/apache/airflow/blob/main/chart/templates/check-values.yaml). I can add these if you think they make sense

[o-nikolas]

Hello @o-nikolas Would it be worth adding a sentence about the CeleryKubernetesExecutor to the update guide https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html#breaking-changes? I just image trying to upgrade the instances I've managed at my previous job that used the CeleryKubernetesExecutor, and I don't think going though the upgrade guide I would have figured out what to do or where to look. I very much like the change, but a pointer to the doc would be appreciated in my view (or a check in the helm templates like https://github.com/apache/airflow/blob/main/chart/templates/check-values.yaml). I can add these if you think they make sense

That's a great idea! I can take the first one (updating the airflow docs) would you like to produce a PR for the latter item of check values @GrumpyCat51?

[GrumpyCat51]

@o-nikolas I'll add them later today