Add virtualization to grid view

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""
DAG for testing Grid View performance with 5000 tasks.

This DAG creates a complex structure with:
- 50 top-level task groups
- Each group contains 100 tasks organized in 10 sub-groups
- Mix of sequential and parallel dependencies
- Total: 5000 EmptyOperator tasks
"""
from airflow.sdk import DAG
from airflow.sdk.definitions.taskgroup import TaskGroup
from airflow.operators.empty import EmptyOperator
from datetime import datetime


with DAG(
    dag_id="grid_view_performance_test_5000_tasks",
    start_date=datetime(2025, 1, 1),
    schedule=None,
    catchup=False,
    tags=["performance", "test", "grid_view"],
    description="Complex DAG with 5000 tasks to test grid view performance",
) as dag:

    # Entry point task
    start = EmptyOperator(task_id="start")

    # Exit point task
    end = EmptyOperator(task_id="end")

    previous_group_final_tasks = [start]

    # Create 50 main task groups
    for main_group_idx in range(50):
        with TaskGroup(group_id=f"main_group_{main_group_idx}") as main_group:

            # Entry task for this main group
            group_start = EmptyOperator(task_id="group_start")

            # Exit task for this main group
            group_end = EmptyOperator(task_id="group_end")

            sub_group_final_tasks = []

            # Create 10 sub-groups within each main group
            for sub_group_idx in range(10):
                with TaskGroup(group_id=f"sub_group_{sub_group_idx}") as sub_group:

                    # Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)
                    sub_tasks = []
                    for task_idx in range(10):
                        task = EmptyOperator(task_id=f"task_{task_idx}")
                        sub_tasks.append(task)

                    # Create dependencies within sub-group
                    # First 5 tasks run in parallel, then next 5 tasks depend on them
                    for i in range(5, 10):
                        sub_tasks[i - 5] >> sub_tasks[i]

                    # Last task of sub-group
                    sub_group_final = sub_tasks[-1]

                # Connect group_start to each sub-group's first tasks
                group_start >> sub_tasks[0]
                group_start >> sub_tasks[1]

                # Collect final tasks from each sub-group
                sub_group_final_tasks.append(sub_group_final)

            # Connect all sub-group final tasks to group_end
            sub_group_final_tasks >> group_end

        # Connect previous groups to current group
        # Create some inter-group dependencies for complexity
        if main_group_idx % 5 == 0:
            # Every 5th group depends on all previous group tasks
            previous_group_final_tasks >> group_start
        else:
            # Other groups depend on the immediately previous group
            previous_group_final_tasks[-1] >> group_start

        # Track this group's final task
        previous_group_final_tasks.append(group_end)

    # Connect all final group tasks to the end task
    previous_group_final_tasks >> end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲