Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Add cursor based pagination for get_task_instances endpoint - #64845

Merged
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances
Apr 16, 2026
Merged

Add cursor based pagination for get_task_instances endpoint#64845
pierrejeambrun merged 11 commits into
apache:mainfrom
astronomer:feature/cursor-pagination-task-instances

Conversation

@pierrejeambrun

@pierrejeambrunpierrejeambrun commented Apr 7, 2026

Copy link
Copy Markdown
Member

related: #62027

Performance improvement. Add API support for the get_task_instances endpoint for 'cursor' based pagination.

At scale, just counting the number of rows is taking too long (60s because the joins prevent using the indexes and trigger parallel full scans, also just a bare count on the table is 2-18s, which is too long). Switching to cursor based pagination will allow to plug the UI on this and reduce this delay. Getting a page takes now <100ms while it was taking 60s on my local setup with 40M TIs.

This keeps backward compatibility, defaulting the list endpoint to 'offset' based pagination. While still allowing passing down a cursor to get a cursor based one.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Cursor] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
@pierrejeambrunpierrejeambrun added this to the Airflow 3.2.1 milestone Apr 7, 2026
@pierrejeambrunpierrejeambrun self-assigned this Apr 7, 2026
@pierrejeambrun
pierrejeambrun marked this pull request as draft April 7, 2026 14:54
@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Apr 7, 2026
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

WIP, needs some cleaning and a few iterations.

Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
@pierrejeambrun
pierrejeambrun marked this pull request as ready for review April 8, 2026 12:05
@pierrejeambrunpierrejeambrun changed the title WIP Cursor pagination task instancesAdd cursor based pagination for get_task_instances endpointApr 8, 2026
Comment threadairflow-core/src/airflow/api_fastapi/common/cursors.py Outdated

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see cursor based pagination though!

@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

I think you're right, this gave me the impression it was working but the actual items returned are not correct on walking backward.

Let me fix it and add a test for it

@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch 2 times, most recently from 78cdb4c to bbe7ea3CompareApril 16, 2026 14:31
@pierrejeambrun
pierrejeambrunforce-pushed the feature/cursor-pagination-task-instances branch from bbe7ea3 to d9c6772CompareApril 16, 2026 15:17
@pierrejeambrun
pierrejeambrun merged commit e11c603 into apache:mainApr 16, 2026
141 checks passed
@pierrejeambrun
pierrejeambrun deleted the feature/cursor-pagination-task-instances branch April 16, 2026 21:37
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 17, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
@pierrejeambrun

Copy link
Copy Markdown
MemberAuthor

Manual backport #65405

@pierrejeambrunpierrejeambrun mentioned this pull request Apr 17, 2026
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Apr 20, 2026
…4845)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (apache#1613, apache#3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 21, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
pierrejeambrun added a commit to pierrejeambrun/airflow that referenced this pull request Apr 22, 2026
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in apache#64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
vatsrahul1001 pushed a commit that referenced this pull request Apr 23, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
pierrejeambrun added a commit that referenced this pull request Apr 23, 2026
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
potiuk pushed a commit that referenced this pull request Apr 26, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request Apr 27, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…65405)
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_task_instances endpoint. Offset pagination remains
the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include
total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of
mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated
common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as
{"type": ..., "value": ...} objects, removing the fragile
column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset
pagination (patch, clear, batch, mapped) so the OpenAPI spec
and generated UI client reflect the correct types. Only
get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness.
Replace flat OR-of-prefix-equalities with nested and/or keyset
predicate for better composite index range scans. Always use
ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes
ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with
a single flat model using optional fields. OpenAPI oneOf + discriminator
is not handled correctly by hey-api/openapi-ts (#1613, #3270): return
types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning
next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor
was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit
instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of
silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c603)
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
…5604) (#65746)
* Add cursor based pagination for get_dag_runs endpoint
Add cursor-based (keyset) pagination as an alternative to offset-based
pagination on the get_dag_runs endpoint. Offset pagination remains the
default and is not deprecated. Mirrors the pattern introduced for
get_task_instances in #64845.
- Response schema adds next_cursor / previous_cursor and makes
total_entries optional (populated only for offset pagination).
- Fetch limit+1 rows to detect last page and return next_cursor=null
when no more results exist.
- Rewire the DagRuns listing page to use cursor pagination via the
DataTable's cursorPagination prop, which eliminates the COUNT(*)
query for large datasets.
* Small Adjustments
* Small Adjustments
(cherry picked from commit c57abb9)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pierrejeambrun@ashb@vatsrahul1001