Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern by justinpakzad · Pull Request #63665 · apache/airflow

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

justinpakzad · 2026-03-15T18:12:18Z

Currently the PATCH /dags endpoint only updates the first page of matching DAGs (default limit of 50), meaning if the filters match more than one page, only the first page gets paused/unpaused. This fixes that by iterating through all pages to collect all matching dag ids before applying the update.

Also added to the docstring to clarify that dag_id_pattern is required to match any dags, and that ~ or % can be passed as wildcards to match all.

closes: #50555

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.5 (mainly for generating test dags)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

pierrejeambrun

This is not needed. I believe you can achieve this by passing ~ or % as a wildcard dag_id_search param.

This can be detailed in the endpoint documentation (docstring that ends up in the openAPI spec) and will avoid accident or adding more code.

Also there is a pagination bug. Only the first matching 25 dags will be unpaused, we need to iterate though all pages.

Do you mind reconverting this PR to fixing the pagination issue so we actually update all the dags that match the query, and also add some docstring to explain the wildcard pattern ?

justinpakzad · 2026-03-28T14:47:58Z

I've update the PR. Just wanted to leave a couple of notes on the approach.

The first call uses the users (or default) offset and limit to fetch the data for the response and the total entries. Then we loop through all entries and get the relevant dag ids that need to be updated. I used the limit as the batch size here (for fetching) as it made the most sense (rather than just specifying a fixed number (e.g, 500)) but happy to change that if there is a preferred or optimal batch size. The reason we iterate over all the dags in the loop instead of using the results from the first call is that the caller's offset may not be 0, so we can't guarantee the first page covers the start of the dataset.

I ran some tests, pausing/unpausing 100+ dags and it's working as expected. I added to the docstring to explain the wildcard pattern and created additional tests cases to test matching any tags, multiple tags, and multiple dags patched.

pierrejeambrun

Thanks for the update. Doc update looks nice

I just realized that there is a better approach than looping the pages for fixing the bug. (details in the comment below)

A couple of suggestion / improvements. And we should be good.

…wildcard usage

…l test case

pierrejeambrun

LGTM, thanks

…63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886)

…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>

boring-cyborgBot added the area:API Airflow's REST/HTTP API label Mar 15, 2026

justinpakzad marked this pull request as ready for review March 15, 2026 19:19

justinpakzad requested review from bugraoz93, choo121600, ephraimbuddy, jason810496, pierrejeambrun, rawwar and shubhamraj-git as code owners March 15, 2026 19:19

justinpakzad force-pushed the fix/dags-patch-pause-with-tags branch from c032f2c to 3b2f945Compare March 15, 2026 19:19

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Mar 17, 2026

pierrejeambrun reviewed Mar 20, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dags.py Outdated
Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py Outdated
Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py

justinpakzad force-pushed the fix/dags-patch-pause-with-tags branch from 3b2f945 to 34995d8Compare March 21, 2026 17:51

justinpakzad requested review from bbovenzi, guan404ming, ryanahamilton and vatsrahul1001 as code owners March 21, 2026 17:51

justinpakzad changed the title ~~Fix: allow PATCH /dags to filter by tags/owners without dag_id_pattern~~Fix: PATCH /dags pagination bug and document wildcard dag_id_patternMar 21, 2026

pierrejeambrun reviewed Apr 1, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py Outdated
Comment threadairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dags.py

justinpakzad force-pushed the fix/dags-patch-pause-with-tags branch from 34995d8 to 7cd8448Compare April 1, 2026 23:50

justinpakzad added 3 commits April 1, 2026 22:47

fixed pagination bug and updated docstring to clarify dag_id_pattern …
44a8144
…wildcard usage

removed batch loop to update all dags in one shot and added additiona…
9ae53ef
…l test case

Fixed MySQL subquery issue
3fa2ec1

justinpakzad force-pushed the fix/dags-patch-pause-with-tags branch from 7cd8448 to 3fa2ec1Compare April 2, 2026 02:48

pierrejeambrun approved these changes Apr 3, 2026
View reviewed changes

pierrejeambrun added this to the Airflow 3.2.1 milestone Apr 3, 2026

pierrejeambrun merged commit 9504886 into apache:mainApr 3, 2026
143 of 145 checks passed

vatsrahul1001 mentioned this pull request Apr 15, 2026
[v3-2-test] Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern #65309
Merged

vatsrahul1001 modified the milestones: Airflow 3.2.1, Airflow 3.2.2 Apr 15, 2026

vatsrahul1001 mentioned this pull request May 18, 2026
[v3-2-test] Fix XCom PATCH/POST to store native values (instead of json.dumps) (#64220) #67116
Merged
1 task