Bundle root should be added to import path

[gbonazzoli]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

If you define a DAG in a git repository and this DAG has to import an internal modules, in my case named Utility:

from Utility.VMwareUtils import vCenter

Airflow gives this error:

The same code works in a normal "dags" directory.

The configuration in airflow.cfg is:

dag_bundle_config_list = [
      {
        "name": "git_repo_dags_veeam",
        "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
        "kwargs": {"tracking_ref": "main", "git_conn_id": "dags_veeam", "refresh_interval": 60}
      }
    ]

What you think should happen instead?

It has to work in the same classical way. Probably it is only a matter to change the default directory from with the Python interpreter execute the code.

How to reproduce

Under a git repository:

• create a directory named Utility
• create a python file named Utility/pippo.py containing a class or function definition
• create a puthon file for a new DAG and import the function defined in Utility/pippo.py

BANG !

Operating System

Ubuntu 24.04.2 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.10.6
apache-airflow-providers-common-compat==1.6.0
apache-airflow-providers-common-io==1.5.4
apache-airflow-providers-common-sql==1.26.0
apache-airflow-providers-git==0.0.2
apache-airflow-providers-postgres==6.1.3
apache-airflow-providers-redis==4.0.2
apache-airflow-providers-smtp==2.0.3
apache-airflow-providers-ssh==4.0.1
apache-airflow-providers-standard==1.0.0

Deployment

Virtualenv installation

Deployment details

Nothing special, vanilla Airflow 3.0.0 in a vanilla Ubuntu 24.04 LXD container

Anything else?

no

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[vatsrahul1001]

@gbonazzoli can you move the utility folder in your dag folder and confirm if this works for you or not?

[gbonazzoli]

@vatsrahul1001

Yes,

coping the Utility folder under the "standard" dags, make airflow import the DAGS correctly:

Bundle               File Path                             PID  Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-------------------  ----------------------------------  -----  ------------------  --------  ----------  ---------------  -------------
git_repo_dags_veeam  VEEAM_CONTROL_BACKUP_POLICY.py        746  0.37s                      0           0
git_repo_dags_veeam  Utility/VMwareUtils.py                747  0.36s                      0           0
git_repo_dags_veeam  VEEAM_EXPORT_FULL_BACKUP.py                                           0           0
git_repo_dags_veeam  VEEAM_BACKUP.py                                                       0           0
dags-folder          Utility/VMwareUtils.py                                                0           0
================================================================================ 

[uranusjr]

We have this line in airflow.settings:

airflow/airflow-core/src/airflow/settings.py

Line 618 in 2d19058

prepare_syspath_for_dags_folder()

that adds the dags folder to the import path, and thus makes everything in there importable. This is not done to any other bundles, so they don’t work the same.

Since people are likely to expect things in a bundle to be importable, we should probably add something to the dag processor, so the root folder of each bundle is on the import path when the bundle is being processed. We should probably also remove this line from settings so things inside the dags folder don’t pollute other bundles, but I’m not sure if it’d have any unintended side effects.

This is not specific to the git bundle logic, but all kinds of bundles (except the one that exposes the dags folder).