Skip to content

Commit

Permalink
Add standalone DAG processor to docker compose (apache#45650)
Browse files Browse the repository at this point in the history
We are moving to only support a standalone DAG processor in Airflow 3,
so let's switch our compose setup to have it. We can remove the env var
once that's the only option, but for now it let's us avoid a massive PR
:)
  • Loading branch information
jedcunningham authored Jan 14, 2025
1 parent f616c62 commit b9d1203
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 0 deletions.
16 changes: 16 additions & 0 deletions docs/apache-airflow/howto/docker-compose/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ x-airflow-common:
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR: 'true'
AIRFLOW__API__AUTH_BACKENDS: >-
airflow.providers.fab.auth_manager.api.auth.backend.basic_auth,airflow.api.auth.backend.session
# yamllint disable rule:line-length
Expand Down Expand Up @@ -148,6 +149,21 @@ services:
airflow-init:
condition: service_completed_successfully

airflow-dag-processor:
<<: *airflow-common
command: dag-processor
healthcheck:
test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob --hostname "$${HOSTNAME}"']
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully

airflow-worker:
<<: *airflow-common
command: celery worker
Expand Down
1 change: 1 addition & 0 deletions docs/apache-airflow/howto/docker-compose/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ This file contains several service definitions:

- ``airflow-scheduler`` - The :doc:`scheduler </administration-and-deployment/scheduler>` monitors all tasks and DAGs, then triggers the
task instances once their dependencies are complete.
- ``airflow-dag-processor`` - The DAG processor parses DAG files.
- ``airflow-webserver`` - The webserver is available at ``http://localhost:8080``.
- ``airflow-worker`` - The worker that executes the tasks given by the scheduler.
- ``airflow-triggerer`` - The triggerer runs an event loop for deferrable tasks.
Expand Down

0 comments on commit b9d1203

Please sign in to comment.