DAGs are missing and deleted in Airflow 2.10.1

[ldacey]

Apache Airflow version

2.10.0

If "Other Airflow 2 version" selected, which one?

2.10.1

What happened?

I updated to Airflow 2.10.1 and the DAGs in my UI started to disappear. I checked the scheduler logs and it showed that the DAGs were not being found and were being deleted. I quickly reverted back to the working 2.10.0 image after that and I don't have the specific logs.

I am syncing the src folder in my repo as the dags_folder which also contains common scripts, hooks, operators, etc. The plugins_folder is backed into the image and only contains things which don't change often, like XCom backend.

Official Helm chart values:

config:
  core:
    dags_folder: /opt/airflow/dags/repo/src
    plugins_folder: /opt/airflow/plugins

dags:
  gitSync:
    enabled: true
    repo: https://gitlab.domain.com/dags.git
    branch: main
    rev: HEAD
    depth: 1
    subPath: src

This is my dags_folder structure:

src/
├── common
│   ├── __init__,py
├── dags
│   ├── __init__.py
│   ├── account_a
│   │   ├── example_a_dag.py
│   │   └── example_b_dag.py
│   └── account_x
├── hooks
│   ├── __init__.py
├── notifiers
│   ├── __init__.py
├── operators
│   ├── __init__.py
├── projects
│   ├── __init__.py
│   ├── account_a
│   │   ├── transform.py
│   │   └── schemas.py
│   └── account_x
└── sensors
    ├── __init__.py

My .airflowignore is in the src folder:

common/
notifiers/
operators/
sensors/
hooks/
projects/

What you think should happen instead?

The DAGs should still be discovered between minor Airflow releases since nothing else changed.

How to reproduce

Hopefully the structure of my setup above suffices.

I am using the Airflow constraints file when installing extras. linux/amd64 is the target platform used and 2.10.1 is the Airflow version I had trouble with.

    - docker build
      --file Dockerfile
      --pull
      --platform "${TARGET_PLATFORM}"
      --cache-from "${RELEASE_IMAGE}"
      --build-arg BUILDKIT_INLINE_CACHE=1
      --build-arg PYTHON_BASE_IMAGE="python:${PYTHON_VERSION}-slim-${DEBIAN_VERSION}"
      --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}"
      --build-arg AIRFLOW_USE_UV="true"
      --build-arg AIRFLOW_UV_VERSION="${AIRFLOW_UV_VERSION}"
      --build-arg AIRFLOW_EXTRAS="async,docker,cncf.kubernetes,ftp,google,google_auth,http,microsoft.azure,odbc,postgres,sftp,ssh,statsd,openlineage"
      --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="${CONSTRAINTS_VERSION}"
      --build-arg INSTALL_MYSQL_CLIENT="false"
      --build-arg INSTALL_MSSQL_CLIENT="true"
      --build-arg INSTALL_POSTGRES_CLIENT="true"
      --build-arg UPGRADE_TO_NEWER_DEPENDENCIES="true"
      --build-arg ADDITIONAL_RUNTIME_APT_DEPS="git graphviz procps vim odbc-postgresql python3-pybind11 python3-pyodbc libgeos-dev"
      --tag "${RELEASE_IMAGE}"
      "."

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jcrobak]

I'm experiencing this issue with 2.10.1, too. I was able to add some logging and it appears that #41433 is the culprit. If the value of airflow config get-value core dags_folder is a symlink, then the parameter dag_directory to deactivate_stale_dags will be the target of the symlink. This means that os.path.commonpath(...) doesn't find a match.

[potiuk]

I'm experiencing this issue with 2.10.1, too. I was able to add some logging and it appears that #41433 is the culprit. If the value of airflow config get-value core dags_folder is a symlink, then the parameter dag_directory to deactivate_stale_dags will be the target of the symlink. This means that os.path.commonpath(...) doesn't find a match.

Ah yeah . That would definitely explain it !

[potiuk]

cc: @utkarsharma2 @ephraimbuddy -> I think we need a quick 2.10.2

[MCubek]

Also experiencing this issue with 2.10.1 instance deployed with the helm chart and using gitsync.
Symptoms include dags going missing while refreshing webserver ui and then, being unable to trigger dag runs and kubernetes executor task faults.
Webserver reported params missing in its stack trace, and the webserver ui reporting dags missing from dag bag when browsing running dags.

I have since reverted back to 2.10.0 after the issues my deployment experienced during the weekend, but will reproduce and send the full error logs if necessary.

[potiuk]

The symlink is the most likely cause I think we will take it from there (thanks @MCubek @jcrobak @ldacey ) - I think it will mean quick 2.10.2 - so we will appreciate -once we fix it, if you test release candidates we will produce.

[ephraimbuddy]

cc: @utkarsharma2 @ephraimbuddy -> I think we need a quick 2.10.2

Yeah. Looking into this

[jcrobak]

here's a quick repro with docker-compose, in case you need it:

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.10.1/docker-compose.yaml' 

sed -e '/\/opt\/airflow\/dags/ s/^#*/#/' < docker-compose.yaml

mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" >| .env 

mkdir dags/example

cat << EOF >>| dags/example/airflow_dag.py
from datetime import datetime, timedelta

from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator

with DAG(
    "joe-tutorial",
    description="A simple tutorial DAG",
    schedule=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=["example"],
) as dag:
            t1 = BashOperator(
        task_id="print_date",
        bash_command="date",
    )

EOF


cat << EOF >> Dockerfile
FROM apache/airflow:2.10.1-python3.12
COPY --chown=airflow:root dags /opt/dags
RUN rmdir /opt/airflow/dags && ln -s /opt/dags/ /opt/airflow/dags

EOF


docker build . -t airflow:2.10.1-repro
echo -e "AIRFLOW_IMAGE_NAME=airflow:2.10.1-repro" >> .env 


docker-compose up

eventually you'll see a log line like:

airflow-scheduler-1  | [2024-09-09T15:23:14.923+0000] {manager.py:548} INFO - DAG joe-tutorial is missing and will be deactivated.
airflow-scheduler-1  | [2024-09-09T15:23:14.927+0000] {manager.py:560} INFO - Deactivated 1 DAGs which are no longer present in file.
airflow-scheduler-1  | [2024-09-09T15:23:14.933+0000] {manager.py:564} INFO - Deleted DAG joe-tutorial in serialized_dag table

[raphaelauv]

same experience with 2.10.1 and 1 dagprocessor

Screencast.from.09-09-2024.17.43.19.webm

[kand617]

I recently upgraded to 2.8 and am seeing a similar thing.
DAG count suddenly goes to 0 and then slowly builds back up... Then after sometime back to 0...
Was just going to upgrade to 2.10 thinking it maybe better.