Skip to content

Conversation

@abhijeets25012-tech
Copy link

This change adds support for deploying separate DAG Processor pods per DAG bundle in the Airflow Helm chart.

  • Introduces deployPerBundle configuration under dagProcessor in values.yaml.
  • Allows independent scaling, resource allocation, and failure isolation per DAG bundle.
  • Uses existing --bundle-name option in Airflow core.
  • Backward compatible: single deployment is used if deployPerBundle.enabled is false.
  • Supports bundle-specific overrides for replicas and resources.

Example configuration:

dagProcessor:
deployPerBundle:
enabled: true
args: ["bash", "-c", "exec airflow dag-processor --bundle-name {{ bundleName }}"]
bundleOverrides:
dags-folder:
replicas: 2
resources:
requests:
memory: "2Gi"
cpu: "1000m"

@jscheffl
Copy link
Contributor

Please open a half-baked PR in draft before it make sense to review.

@jscheffl jscheffl marked this pull request as draft January 25, 2026 21:45
@uplsh580
Copy link
Contributor

I'm working on this.
#61039

Related Issue: #61037

@jedcunningham
Copy link
Member

Duplicate of #61039.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants