feat: implement patching of task group instances in API by OscarLigthart · Pull Request #62812 · apache/airflow

OscarLigthart · 2026-03-03T19:28:43Z

Context

In Airflow 3 the ability to mark a full task group as failed or success is currently missing. In this PR, I try to implement the logic into the API, that can then be called from the frontend to ensure the functionality returns.

I deliberately split the feature in two separate PRs, so the reviewing process can be more targeted to the individually touched components. Should this PR get merged, I will continue to build the remaining requirements into the frontend.

There is an already open PR here: #60161

However, it looks to be stale, and I would love to get this feature into the Airflow 3.2 release.

Implementation

I make use of the BulkTaskInstanceService to perform the state updates with this endpoint. Using it, the implementation should be pretty straightforward :).

Issues

related: #56103

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Opus following the guidelines

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

pierrejeambrun

Thanks for the PR.

Can you add query guards to tests (assert_queries_count) . I'm afraid this will generate N+1 queries problem -> Number of db request will scale linearly with the number of TIs in the group. And we should rework the code not do do that.

Also I find the code overly complicated with numerous duplicated call made at different abstraction level because of function nesting, it makes the whole thing hard to understand and probably sub optimal.

OscarLigthart · 2026-03-09T22:31:41Z

Thanks for the PR.
Can you add query guards to tests (assert_queries_count) . I'm afraid this will generate N+1 queries problem -> Number of db request will scale linearly with the number of TIs in the group. And we should rework the code not do do that.
Also I find the code overly complicated with numerous duplicated call made at different abstraction level because of function nesting, it makes the whole thing hard to understand and probably sub optimal.

Thanks for the detailed review! I think the points made a lot of sense. Apologies for this oversight on my side. I tried following the implementation of the other PR more, removing the BulkTaskInstanceService usage and solving the N+1 query problem.

I still opted to keep them in separate endpoints for clarity. Let me know if you disagree! Also happy to cast this aside and wait for the other PR to be picked up again, if that is more in line with your preference.

One question regarding the assert_queries_count, I've create multiple cases where they remain the same, effectively eliminating the query problem, but I'm guessing there's many things that can impact this amount. In turn, this would break the tests implemented here. Is there a specific rule to keep in mind when applying it? Or is just setting it in the way I do in the test correct?

EDIT - I'm now asserting whether the query count is equal between different size task groups. Should solve the above 👍

potiuk · 2026-03-11T12:32:29Z

@OscarLigthart Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 64 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.
⚠️Unresolved review comments: This PR has 2 unresolved review threads from maintainers. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See pull request guidelines.

Note: Your branch is 64 commits behind main. Please rebase and push again to get up-to-date CI results.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

potiuk · 2026-04-23T01:09:11Z

@OscarLigthart Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

❌ Unresolved review comments (5 threads (from maintainers)): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Once you believe a thread is resolved — whether by pushing a fix or by explaining why the suggestion doesn't apply — please mark it as resolved yourself by clicking the 'Resolve conversation' button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as 'still waiting on the author' and keeps the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

OscarLigthart · 2026-04-23T09:06:25Z

Putting back to ready for review as I'm waiting for maintainer comments in unresolved threads.

potiuk · 2026-04-27T01:35:10Z

@OscarLigthart — There are 2 unresolved review threads on this PR from @pierrejeambrun. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

potiuk · 2026-04-27T01:37:02Z

Hmmm.. I see that those things are resolved .... Hmm... Probably we should improve the automation.

potiuk · 2026-04-27T01:42:05Z

@pierrejeambrun -> I will improve the automation here, but this one looks like it's resolved and I marked it as ready for review + invite you to re-review :)

potiuk · 2026-04-27T01:43:30Z

sorry @OscarLigthart for a bit of spam :( ... we are improving things.

potiuk · 2026-04-27T02:02:04Z

#65916 @pierrejeambrun

OscarLigthart · 2026-04-27T11:21:54Z

No worries at all, happy to be a guinea pig. Also very much understand the need for this automation.

pierrejeambrun

I rebased the branch, did some small adjustments. Looking good.

* feat: implement patching of task group instances in API * feat: improve upon N+1 query problem * fix: revert back to multi calls * feat: implement set_task_group_state with locking mechanism * feat: some cleanup and additional tests * chore: improve function name * Small Adjustments * Small Adjustments * Fix note bug --------- Co-authored-by: pierrejeambrun <pierrejbrun@gmail.com>

jose-padron-momox · 2026-06-11T06:51:59Z

being able to mark a whole task group as success/failed in the UI is something we really miss from Airflow 2.x. Thank you to everyone working on it! Glad to see the backend support (#62812) merged; looking forward to the UI follow-up landing too.

pierrejeambrun · 2026-06-11T10:00:27Z

@jose-padron-momox front-end PR was merged here:

#66146

That will be available in 3.3.0 release coming soon.

OscarLigthart requested review from bbovenzi, bugraoz93, choo121600, dheerajturaga, ephraimbuddy, guan404ming, jason810496, pierrejeambrun, potiuk, rawwar, ryanahamilton, shubhamraj-git and vatsrahul1001 as code owners March 3, 2026 19:28

boring-cyborgBot added area:airflow-ctl area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Mar 3, 2026

OscarLigthart force-pushed the feat/mark-task-group-status branch from c03d03f to 60004daCompare March 3, 2026 21:06

pierrejeambrun reviewed Mar 9, 2026
View reviewed changes

OscarLigthart force-pushed the feat/mark-task-group-status branch 6 times, most recently from 767a96c to 85e4f72Compare March 10, 2026 17:21

potiuk marked this pull request as draft March 11, 2026 12:32

OscarLigthart force-pushed the feat/mark-task-group-status branch from 85e4f72 to 0bfc970Compare March 12, 2026 22:26

OscarLigthart marked this pull request as ready for review March 12, 2026 22:46

OscarLigthart force-pushed the feat/mark-task-group-status branch from cabb86e to 88add8cCompare March 13, 2026 10:10

OscarLigthart force-pushed the feat/mark-task-group-status branch from 1cfe416 to 326b655Compare April 22, 2026 10:34

potiuk marked this pull request as draft April 23, 2026 01:09

OscarLigthart marked this pull request as ready for review April 23, 2026 09:05

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 27, 2026

OscarLigthartand others added 7 commits April 29, 2026 13:09

feat: implement patching of task group instances in API
7555655

feat: improve upon N+1 query problem
bb966e8

fix: revert back to multi calls
b2063e6

feat: implement set_task_group_state with locking mechanism
fd45699

feat: some cleanup and additional tests
c066373

chore: improve function name
1090679

Small Adjustments
a0c6f89

pierrejeambrun force-pushed the feat/mark-task-group-status branch from 326b655 to a0c6f89Compare April 29, 2026 11:14

pierrejeambrun mentioned this pull request Apr 29, 2026
Restore ability to mark task groups as success/failed in UI #60161
Closed

pierrejeambrun added 2 commits April 29, 2026 15:09

Small Adjustments
38e7a74

Fix note bug
f36c9fc

pierrejeambrun added this to the Airflow 3.3.0 milestone Apr 29, 2026

pierrejeambrun approved these changes Apr 29, 2026
View reviewed changes

bbovenzi merged commit 9a09bc2 into apache:mainApr 29, 2026
140 checks passed

pierrejeambrun mentioned this pull request Apr 30, 2026
UI: Mark task group as success/failed #66146
Merged
1 task

Uh oh!

Conversation

OscarLigthart commented Mar 3, 2026

Context

Implementation

Issues

Was generative AI tooling used to co-author this PR?

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OscarLigthart commented Mar 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Mar 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Apr 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OscarLigthart commented Apr 23, 2026

Uh oh!

potiuk commented Apr 27, 2026

Uh oh!

potiuk commented Apr 27, 2026

Uh oh!

potiuk commented Apr 27, 2026

Uh oh!

potiuk commented Apr 27, 2026

Uh oh!

potiuk commented Apr 27, 2026

Uh oh!

OscarLigthart commented Apr 27, 2026

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jose-padron-momox commented Jun 11, 2026

Uh oh!

pierrejeambrun commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pierrejeambrun left a comment •
edited
Loading

OscarLigthart commented Mar 9, 2026•
edited
Loading

potiuk commented Mar 11, 2026•
edited
Loading

potiuk commented Apr 23, 2026•
edited
Loading

pierrejeambrun left a comment •
edited
Loading