Combined Executor - KubernetesExecutor + LocalExecutor

[hendoxc]

Description
In short: There exists already combined executor support for Celery + Kubernetes, I think that also adding Local + Kubernetes
would be warmly welcomed.

Use case / motivation

Many tasks are low-resource intensive tasks on the airflow pod, It makes sense to keep things running on LocalExecutor, 0 overhead. However I also have a large number of dags, that have some long running tasks which are somewhat resource hungry, for these I currently use KubernetesPodOperator, this makes the Dags more scalable and overall workload elastic. Being able to easily switch between easily between KubernetesExecutor and LocalExecutor, just feels natural in this sense.

Related Issues

N/A
@potiuk

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Worth looking at creating it !

[acmh]

@potiuk Can I grab this issue? If yes, where I can start looking?

[potiuk]

Take a look at CeleryKubernetesExecutor. Although this one would require a little more intrinsic knowledge of Airflow - so please take a little more knowledge of Airflow - like integration with Helm Chart etc. Maybe marking it as first-good-issue was not the best idea :) . Just take a look at the C-K executor and see if this is something you would like to take on.

[subkanthi]

Take a look at CeleryKubernetesExecutor. Although this one would require a little more intrinsic knowledge of Airflow - so please take a little more knowledge of Airflow - like integration with Helm Chart etc. Maybe marking it as first-good-issue was not the best idea :) . Just take a look at the C-K executor and see if this is something you would like to take on.

Is this being worked on, Im trying to find an example DAG of how to mark the tasks as kubernetes_queue, would be helpful if you can point to it - @potiuk

[subkanthi]

Actually @potiuk can u assign this to me please, if no one else is working.

[potiuk]

Feel free!