Error cloning repository

[anavrotski]

Apache Airflow version

3.1.7

If "Other Airflow 3 version" selected, which one?

3.1.6

What happened?

We are observing a sporadic issue (so I don't know how to reproduce) with GitDagBundles when cloning DAG from self-hosted GitLab (see log below), and this error obviously is not raising an alert by calling DAG callback, so it is "hidden" in some sense.

[2026-02-10 06:05:02] WARNING - Repository clone/open failed, cleaning up and retrying bundle_name=etl-dags-bundle version=09b6... bare_repo_path=PosixPath('/tmp/airflow/dag_bundles/.../bare') repo_path=PosixPath('/tmp/airflow/dag_bundles/.../versions/09b6...') versions_path=PosixPath('/tmp/airflow/dag_bundles/etl-dags-bundle/versions') git_conn_id=git_conn submodules= exc=InvalidGitRepositoryError('/tmp/airflow/dag_bundles/.../versions/09b6...') source=airflow.providers.git.bundles.git loc=git.py:180
[2026-02-10 06:05:02] INFO - Cloning repository bundle_name=... version=09b6... bare_repo_path=PosixPath('/tmp/airflow/dag_bundles/.../bare') repo_path=PosixPath('/tmp/airflow/dag_bundles/.../versions/09b6...') versions_path=PosixPath('/tmp/airflow/dag_bundles/.../versions') git_conn_id=git_conn submodules= source=airflow.providers.git.bundles.git loc=git.py:166
[2026-02-10 06:05:02] WARNING - Repository clone/open failed, cleaning up and retrying bundle_name=... version=09b6... bare_repo_path=PosixPath('/tmp/airflow/dag_bundles/.../bare') repo_path=PosixPath('/tmp/airflow/dag_bundles/.../versions/09b6...') versions_path=PosixPath('/tmp/airflow/dag_bundles/.../versions') git_conn_id=git_conn submodules= exc=GitCommandError(['git', 'clone', '-v', '--', '/tmp/airflow/dag_bundles/.../bare', '/tmp/airflow/dag_bundles/.../versions/09b6...'], 128, b"fatal: destination path '/tmp/airflow/dag_bundles/.../versions/09b6...' already exists and is not an empty directory.\n") source=airflow.providers.git.bundles.git loc=git.py:180
[2026-02-10 06:05:02] ERROR - Top level error source=task loc=task_runner.py:1562
RuntimeError: Error cloning repository
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1550 in main
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 732 in startup
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 618 in parse
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/git/bundles/git.py", line 155 in initialize
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/git/bundles/git.py", line 126 in _initialize
GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v -- /tmp/airflow/dag_bundles/.../bare /tmp/airflow/dag_bundles/.../versions/09b6...
  stderr: 'fatal: destination path '/tmp/airflow/dag_bundles/.../versions/09b6...' already exists and is not an empty directory.
'
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/git/bundles/git.py", line 124 in _initialize
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338 in wrapped_f
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477 in __call__
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378 in iter
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 420 in exc_check
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 187 in reraise
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449 in result
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401 in __get_result
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480 in __call__
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/git/bundles/git.py", line 169 in _clone_repo_if_required
File "/home/airflow/.local/lib/python3.12/site-packages/git/repo/base.py", line 1538 in clone_from
File "/home/airflow/.local/lib/python3.12/site-packages/git/repo/base.py", line 1409 in _clone
File "/home/airflow/.local/lib/python3.12/site-packages/git/util.py", line 512 in finalize_process
File "/home/airflow/.local/lib/python3.12/site-packages/git/cmd.py", line 419 in wait
[2026-02-10 06:05:02] WARNING - Process exited abnormally exit_code=1 source=task

What you think should happen instead?

I think that in this Git provider's file, the @retry decorator from tenacity should be supplemented with parameters such as wait, wait_exponential, or wait_fixed to introduce a small delay between retries. As you can see in the log above, the current retries occur less than 1 second apart, which causes an issue.

How to reproduce

I do not know.

Operating System

linux

Versions of Apache Airflow Providers

git provider v 0.2.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

official helm chart 1.18

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dominikhei]

This looks like some kind of race condition. Having a retry delay is an option, I think if we could implement some kind of lock this would be even better. I can have a look at it if deemed necessary.

[anavrotski]

This looks like some kind of race condition. Having a retry delay is an option, I think if we could implement some kind of lock this would be even better. I can have a look at it if deemed necessary.

Not sure why, but I'm seeing this "WARNING - Repository clone/open failed, ..." in almost every DAG run.

[dominikhei]

This looks like some kind of race condition. Having a retry delay is an option, I think if we could implement some kind of lock this would be even better. I can have a look at it if deemed necessary.

Not sure why, but I'm seeing this "WARNING - Repository clone/open failed, ..." in almost every DAG run.

WARNING - Repository clone/open failed, cleaning up and retrying The cleanup does not work as we have the error: stderr: 'fatal: destination path '/tmp/airflow/dag_bundles/.../versions/09b6...' already exists and is not an empty directory. I am not sure how the git operator works, will have a look at it, but to me it looks like the cloning is tried before everything is cleaned up and thus either incomplete or not synchronized with the retry.

[Arunodoy18]

Thanks for the detailed investigation!

From the discussion and logs, this does look like a race between cleanup and clone operations across concurrent runs.

I’m interested in working on this. My initial thought is to explore adding version-level locking around clone/open operations to ensure only one process prepares a bundle version directory at a time, instead of relying on retries.

If that direction sounds reasonable, I’d be happy to prototype a fix and add concurrency tests.

Please let me know if you have any preferences on approach.

[ephraimbuddy]

This looks like some kind of race condition. Having a retry delay is an option, I think if we could implement some kind of lock this would be even better. I can have a look at it if deemed necessary.

Not sure why, but I'm seeing this "WARNING - Repository clone/open failed, ..." in almost every DAG run.

The particular case you mentioned is being addressed here: #61847