Add virtualization to grid view

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲

[bbovenzi]

Virtualize the grid view to improve rendering performance on very large dags.

Before (10 dag runs with 1000 tasks):

After (10 dag runs with 1000 tasks):

Results go from unusable to slow. I'll look for other improvements in follow up PRs. Notably some of the API calls still take 1-3 seconds.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

here's my dag

"""DAG for testing Grid View performance with 5000 tasks.This DAG creates a complex structure with:- 50 top-level task groups- Each group contains 100 tasks organized in 10 sub-groups- Mix of sequential and parallel dependencies- Total: 5000 EmptyOperator tasks"""fromairflow.sdkimportDAGfromairflow.sdk.definitions.taskgroupimportTaskGroupfromairflow.operators.emptyimportEmptyOperatorfromdatetimeimportdatetimewithDAG(
dag_id="grid_view_performance_test_5000_tasks",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
tags=["performance", "test", "grid_view"],
description="Complex DAG with 5000 tasks to test grid view performance",
) asdag:
# Entry point taskstart=EmptyOperator(task_id="start")
# Exit point taskend=EmptyOperator(task_id="end")
previous_group_final_tasks= [start]
# Create 50 main task groupsformain_group_idxinrange(50):
withTaskGroup(group_id=f"main_group_{main_group_idx}") asmain_group:
# Entry task for this main groupgroup_start=EmptyOperator(task_id="group_start")
# Exit task for this main groupgroup_end=EmptyOperator(task_id="group_end")
sub_group_final_tasks= []
# Create 10 sub-groups within each main groupforsub_group_idxinrange(10):
withTaskGroup(group_id=f"sub_group_{sub_group_idx}") assub_group:
# Create 10 tasks per sub-group (10 sub-groups * 10 tasks = 100 tasks per main group)sub_tasks= []
fortask_idxinrange(10):
task=EmptyOperator(task_id=f"task_{task_idx}")
sub_tasks.append(task)
# Create dependencies within sub-group# First 5 tasks run in parallel, then next 5 tasks depend on themforiinrange(5, 10):
sub_tasks[i-5] >>sub_tasks[i]
# Last task of sub-groupsub_group_final=sub_tasks[-1]
# Connect group_start to each sub-group's first tasksgroup_start>>sub_tasks[0]
group_start>>sub_tasks[1]
# Collect final tasks from each sub-groupsub_group_final_tasks.append(sub_group_final)
# Connect all sub-group final tasks to group_endsub_group_final_tasks>>group_end# Connect previous groups to current group# Create some inter-group dependencies for complexityifmain_group_idx%5==0:
# Every 5th group depends on all previous group tasksprevious_group_final_tasks>>group_startelse:
# Other groups depend on the immediately previous groupprevious_group_final_tasks[-1] >>group_start# Track this group's final taskprevious_group_final_tasks.append(group_end)
# Connect all final group tasks to the end taskprevious_group_final_tasks>>end

[bbovenzi]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

[pierrejeambrun]

Super cool! Tested locally and working as expected for me.

[dheerajturaga]

@bbovenzi, Im seeing for large dags the grid view task alignment is broken in this PR

Thanks. What OS+browser are you on?

chrome with airflow running on ubuntu (wsl) via breeze

[bbovenzi]

@dheerajturaga Fixed

[dheerajturaga]

Thanks @bbovenzi , for implementing this! Very Cool! this is definitely a very big improvement (I'm seeing 4x improvement 8s -> 2s)

Old:

new:

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[kaxil]

#protm

[jscheffl]

Wow. Impressive!

[choo121600]

Wow Coool!😲