improve grid/ti_summaries and grid/runs

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement

[henry3260]

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

What

This improves the Grid API performance for DAGs with large mapped task instance cardinality.

There are two main changes:

1. grid/runs no longer loads all task_instances / task_instances_histories for every DagRun on the page just to compute dag_versions.
   Instead, it fetches the paginated runs first and then performs a slimmer lookup for the distinct DAG versions needed for the response.

2. grid/ti_summaries no longer builds and propagates full per-task-instance details lists while aggregating task groups.
   It now keeps only the summary data needed for the response, such as child state counts and min/max dates.

Why

The previous implementation scaled poorly for DAGs that combine:

• deep TaskGroup nesting
• large dynamic task mapping
• many task instances per run

In particular, grid/ti_summaries was keeping details during aggregation even though those details are not part of the response model. This meant we were paying extra CPU and memory cost for intermediate data that was never returned to the client.

I don't think we should keep details in the summary path, because the response does not include them. Keeping them only increases temporary allocations and repeated copying when task-group summaries are rolled up.

Result

This keeps the API contract unchanged, but reduces unnecessary ORM loading and Python-side aggregation work.

On a local benchmark with 5 DAG runs and about 4002 task instances per run:

• grid/runs median latency improved from ~2955 ms to ~43 ms
• grid/runs tracemalloc peak dropped from ~104.6 MiB to ~1.7 MiB
• grid/ti_summaries tracemalloc peak dropped from ~5.4 MiB to ~1.7 MiB

closes: #63975

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[pierrejeambrun]

Thanks for the PR.

Performance gain looks super promising. 🎉

Just a couple of suggestion before we can go ahead.

[Copilot]

Pull request overview

Improves performance and memory usage of the Grid UI FastAPI endpoints (/grid/runs and /grid/ti_summaries) for DAGs with deep TaskGroup nesting and high mapped task-instance cardinality, while keeping the API contract unchanged.

Changes:

• Refactors /grid/runs to avoid eager-loading all task instances / history rows just to compute dag_versions, using a slimmer post-fetch preloading step instead.
• Refactors /grid/ti_summaries aggregation to use a compact summary object (GridNodeAgg) and avoid building/propagating per-TI details lists during TaskGroup rollups.
• Adds a unit test covering the “multiple DAG versions per run” response shape for /grid/runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py	Adds regression test for multiple dag_versions per run in /grid/runs.
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py	Introduces GridNodeAgg and updates aggregate building to avoid propagating TI details.
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py	Updates Grid routes to use slim dag-version prefetching for runs and streaming/compact TI summary aggregation.

[pierrejeambrun]

Nice, looking good, thanks.

[pierrejeambrun]

Unrelated CI failure, merging.

[bbovenzi]

#protm another big improvement to the grid view!

[vatsrahul1001]

@bbovenzi do we need this in 3.2.2? or ok to be moved to 3.3.0 as its an improvement