Tasks in a DAG cannot be executed in parallel to find a solution

[linwanlong]

Apache Airflow version

3.0.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

This is my environmental information

Apache Airflow
version                | 3.0.3                                                         
executor               | LocalExecutor

Here's my config information

[core]
default_timezone = utc
executor = LocalExecutor
parallelism = 32
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
max_consecutive_failed_dag_runs_per_dag = 0

Here's my dag file

from airflow.sdk import dag, task
import time
from airflow.providers.standard.operators.empty import EmptyOperator


@dag(schedule=None, catchup=False, tags=["verification"], max_active_tasks=10)
def busy_box():

    @task
    def task_a():
        print("start task A")
        time.sleep(10)
        print("end task A")

    @task
    def task_b():
        print("start task B")
        time.sleep(10)
        print("end task B")

    @task
    def task_c():
        print("start task c")
        time.sleep(10)
        print("end task c")

    start = EmptyOperator(task_id="start")
    a = task_a()
    b = task_b()
    c = task_c()
    end = EmptyOperator(task_id="end")
    # a >> b  # 串行
    # 或者改成
    start >> [a, b, c] >> end  # 并行


busy_box_dag = busy_box()

Present the results

Task C does not start until after the end of Task B. Sometimes the start time of task A and task B is different (task B starts after task A ends), Task C keeps queuing , but this problem is not encountered in Airflow2

This is about the process of airflow3

root     2655981  0.0  0.2 128088 48592 pts/3    Sl+  21:28   0:00 /root/miniconda3/envs/xxx/bin/python /root/.vscode-server/extensions/ms-python.black-formatter-2025.2.0/bundled/tool/lsp_server.py --stdio
root     2656190  0.1  0.9 190396 150140 pts/0   S+   21:28   0:02 airflow api_server -- host:0.0.0.0 port:8080
root     2656199  0.0  0.0  16752 10360 pts/0    S+   21:28   0:00 /root/miniconda3/envs/xxxbin/python3.10 -c from multiprocessing.resource_tracker import main;main(7)
root     2656200  0.3  1.4 487120 230024 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/xxx/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=10) --multiprocessing-fork
root     2656201  0.3  1.4 488744 232272 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/xxx/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=14) --multiprocessing-fork
root     2656202  0.3  1.3 548744 221488 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/xxx/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=18) --multiprocessing-fork
root     2656203  0.3  1.3 551072 223920 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/xxx/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=24) --multiprocessing-fork
root     2656706  3.1  1.0 212584 171700 pts/4   S+   21:31   0:46 /root/miniconda3/envs/xxx/bin/python3.10 /root/miniconda3/envs/xxx/bin/airflow scheduler
systemd+ 2656711  1.0  0.1 226844 22560 ?        Ss   21:31   0:15 postgres: postgres airflow 172.20.0.1(55568) idle
root     2656786  0.8  1.0 279012 166460 pts/6   Sl+  21:31   0:13 /root/miniconda3/envs/xxx/bin/python3.10 /root/miniconda3/envs/xxx/bin/airflow dag-processor
root     2657018  0.0  0.9 204676 147136 pts/4   S+   21:31   0:00 airflow worker -- LocalExecutor: <idle>
root     2657019  0.0  0.9 205468 147964 pts/4   S+   21:31   0:00 airflow worker -- LocalExecutor: <idle>
root     2657511  0.7  0.9 200436 159288 pts/8   S+   21:32   0:09 /root/miniconda3/envs/xxx/bin/python3.10 /root/miniconda3/envs/xxx/bin/airflow triggerer
root     2657555  0.1  0.8 198196 135840 pts/8   S+   21:32   0:02 /root/miniconda3/envs/xxx/bin/python3.10 /root/miniconda3/envs/xxx/bin/airflow triggerer

What you think should happen instead?

I think it could be an issue with the airflow scheduler , this issue did not occur before Airflow2 migrated to Airflow3

How to reproduce

Follow the official Quick Start command and reproduce using the dag file provided above

Operating System

uname_result(system='Linux', node='DESKTOP-QKK3UA9', release='5.15.167.4-microsoft-standard-WSL2', version='#1 SMP Tue Nov 5 00:21:55 UTC 2024', machine='x86_64')

Versions of Apache Airflow Providers

Providers info
apache-airflow-providers-amazon | 9.9.0
apache-airflow-providers-common-compat | 1.7.2
apache-airflow-providers-common-io | 1.6.1
apache-airflow-providers-common-sql | 1.27.3
apache-airflow-providers-fab | 2.3.0
apache-airflow-providers-http | 5.3.2
apache-airflow-providers-smtp | 2.1.1
apache-airflow-providers-standard | 1.4.0

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[yokwejuste]

Please kindly check your configurations on the airflow.cfg file to be sure of your parallelism configs.

parallelism = 32

max_active_tasks_per_dag = 16

dags_are_paused_at_creation = True

max_active_runs_per_dag = 16

max_consecutive_failed_dag_runs_per_dag = 0

Mostly the reasons might be:

Cause	Description
Concurrency limit	Not enough slots (max_active_tasks_per_dag, parallelism, pools)
Executor type	Sequential executor only allows one task at a time
Worker resources	Not enough workers, CPUs, or memory
Queue limits	Limited workers in a queue
Dependency	Upstream not finished, trigger rules not met
Manual actions	Tasks/dag paused, or waiting for external trigger
Scheduler lag	Scheduler is overloaded or slow

[potiuk]

I tried to reproduce it with your DAG and could not.

You likely have a problem with configuration of your system. Maybe you have too little resources (memory/cpu) allocated on your WSL2 instance. Or maybe you have some other WSL2-specific limitations (for example filesystem limitations). Note that airflow only works on fully POSIX-compliant installations - including Operating System and filesystems it uses, so certain combination of those factors might have influenced behaviour of your installation as well.

For example if you are using sqlite db and your filesystem is mounted from Windows, I guess this might heavily influence parallelism of database operations.

My suggestion is to give plenty of resources to your WSL2, make absolutely sure that you install and use airflow installed only inside your linux WSL2 including all filesystem (reinstalling things from scratch might help).

Converting to a discussion.