Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move DAG bundle config into config, not db #44924

Merged
merged 10 commits into from
Dec 17, 2024

Conversation

jedcunningham
Copy link
Member

This moves the DAG bundle config into the Airflow config, instead of being in the db. This:

  • makes it much easier to configure a fresh Airflow instance - no api/cli calls required
  • avoids some security concerns by ensuring only deployment managers, with direct access to the instance, can configure these

The primary downside is this does mean you cannot reconfigure an existing bundle in a running Airflow instance.

airflow/config_templates/config.yml Show resolved Hide resolved
airflow/models/dagbundle.py Outdated Show resolved Hide resolved
airflow/config_templates/config.yml Outdated Show resolved Hide resolved
airflow/dag_processing/bundles/manager.py Show resolved Hide resolved
airflow/models/dag.py Show resolved Hide resolved
jedcunningham and others added 6 commits December 16, 2024 13:55
This moves the DAG bundle config into the Airflow config, instead of
being in the db. This:

- makes it much easier to configure a fresh Airflow instance - no
  api/cli calls required
- avoids some security concerns by ensuring only deployment managers,
  with direct access to the instance, can configure these

The primary downside is this does mean you cannot reconfigure an
existing bundle in a running Airflow instance.
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
@jedcunningham
Copy link
Member Author

Remaining failure is unrelated.

@jedcunningham jedcunningham merged commit 4000905 into apache:main Dec 17, 2024
95 of 96 checks passed
@jedcunningham jedcunningham deleted the bundles_from_config branch December 17, 2024 21:59
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
This moves the DAG bundle config into the Airflow config, instead of
being in the db. This:

- makes it much easier to configure a fresh Airflow instance - no
  api/cli calls required
- avoids some security concerns by ensuring only deployment managers,
  with direct access to the instance, can configure these

The primary downside is this does mean you cannot reconfigure an
existing bundle in a running Airflow instance.
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Jan 6, 2025
This moves the DAG bundle config into the Airflow config, instead of
being in the db. This:

- makes it much easier to configure a fresh Airflow instance - no
  api/cli calls required
- avoids some security concerns by ensuring only deployment managers,
  with direct access to the instance, can configure these

The primary downside is this does mean you cannot reconfigure an
existing bundle in a running Airflow instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-66: DAG Bundle/Manifest area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation
Projects
Development

Successfully merging this pull request may close these issues.

3 participants