Add support for multiple Celery worker groups with queue-specific configurations in Helm chart

[HsiuChuanHsu]

Description

We are using Apache Airflow with the Celery Executor and need to configure the system to allow tasks in different queues to be executed on specific nodes.

The goal is to achieve fine-grained control over task distribution by assigning queues to designated worker nodes.

Use case/motivation

Assign each queue to run on specific worker nodes to optimize resource allocation and task execution.

1. Celery queue groups for different queues and node assignments
2. Each worker group creates a separate Deployment assigned to specific nodes and queues

Use case:

1. Node-specific scheduling: Route certain queues to specialized node pools (GPU, high-memory, etc.)
2. High-priority vs. low-priority tasks: Allocate more resources to critical queues
3. Resource-intensive workloads: Dedicate workers with higher memory/CPU to specific queues

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashb]

For future: no need to create an issue if you are going to open a PR -- you can create the PR with a suitable description straight away.

[miqm]

FYI I stumbled on an old issue requesting similar thing: #34219. Perhaps we should mark it as duplicate and reference this one or mark PR to close both of them.