API Add consuming and producing asset filters on list DagRuns by valentinDruzhinin · Pull Request #56022 · apache/airflow

valentinDruzhinin · 2025-09-24T03:55:25Z

Relates to: #53046

The new consumed_assets and produced_assets fields were added.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

RoyLee1224 · 2025-09-24T14:25:48Z

Hi @valentinDruzhinin , thanks for your work on this!

It looks like we might have been working on a similar issue in parallel. My PR #55735 also works for Dag run view tab, which seems to have a significant overlap with your work here.

Perhaps the maintainers could advise on the best way to combine or move forward with these changes. Thanks!

valentinDruzhinin · 2025-09-24T14:52:47Z

Hi @valentinDruzhinin , thanks for your work on this!
It looks like we might have been working on a similar issue in parallel. My PR #55735 also works for Dag run view tab, which seems to have a significant overlap with your work here.
Perhaps the maintainers could advise on the best way to combine or move forward with these changes. Thanks!

Hey @RoyLee1224 . Thanks for letting me know. It's sad to see that. I hope we can combine our work together.

bbovenzi · 2025-09-24T15:38:28Z

Yes, let's definitely move forward with @RoyLee1224's UX changes. But perhaps we could have this PR focus just on backend changes?

I'm also hesitant about eagerly adding indexes.

valentinDruzhinin · 2025-09-24T15:40:58Z

Yes, let's definitely move forward with @RoyLee1224's UX changes. But perhaps we could have this PR focus just on backend changes?
I'm also hesitant about eagerly adding indexes.

Sounds good. Let me revert the UI changes then. I'll only add the backend changes.

pierrejeambrun

@valentinDruzhinin #55735 PR is very close to be in a mergeable state.

I'm sorry but this PR will most probably not be merged in its current state, in favor of #55735. As Brent suggested we can focus this PR on the backend change that will allow filtering on the consuming assets and producing assets. This is a big enough change to be in a separate PR.

@RoyLee1224, please make sure to communicate when you start working on an issue that already has an assignee, just to avoid doing the work twice. 🙂

RoyLee1224 · 2025-09-24T16:54:13Z

Apologies for the confusion—I was assigned to #53043 and didn't see the overlap with #53046.

@valentinDruzhinin, sorry for stepping on your toes, I'll be more careful about checking for related issues in the future.

valentinDruzhinin · 2025-09-30T16:58:42Z

Hey @pierrejeambrun can you reopen the PR please? I've added the changes on backend what you asked for consuming assets and producing assets. These changes haven't been added at #55735 .
Thank you

pierrejeambrun · 2025-10-01T10:00:51Z

Hey @pierrejeambrun can you reopen the PR please? I've added the changes on backend what you asked for consuming assets and producing assets. These changes haven't been added at #55735 .
Thank you

Cool, let me just update the PR description then

valentinDruzhinin · 2025-10-03T21:15:34Z

@bbovenzi @pierrejeambrun could you please take a look ? :)

bugraoz93

Thanks for the PR! API side looks good! I would appreciate another view from the UI side

valentinDruzhinin · 2025-10-08T16:18:17Z

Thanks for the PR! API side looks good! I would appreciate another view from the UI side

Thanks, I can add UI filters after this PR

bbovenzi · 2025-10-09T19:03:04Z

Yes, UI changes can be done in another PR but I don't see the AssetSummary showing up in the DagRunResponse in typescript. Could you take a look?

bbovenzi · 2025-10-09T19:03:43Z

Also, can we please update the PR description and screenshots to be reflect the latest changes?

valentinDruzhinin · 2025-10-10T20:14:35Z

@bbovenzi Thank you! Fixed the model to show the new fields in response.
And updated the PR description as well!
Please take a look :)

valentinDruzhinin · 2025-10-15T15:25:28Z

@bbovenzi @pierrejeambrun please take a look

jason810496

Nice! Thanks for the PR.

jason810496 · 2025-10-17T04:42:13Z

 )


+class AssetSummary(PydanticBaseModel):


May I ask why do we need PydanticBaseModel instead of BaseModel here ?

jason810496 · 2025-10-17T04:49:41Z

- total_entries=total_entries,
- )
+ dag_runs = list(session.scalars(dag_runs_select))
+


It would be nice to modularize the post processing logic for binding dr.consumed_assets and dr.produced_assets in airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py

jason810496 · 2025-10-17T04:51:51Z


 def _iter_dags() -> Iterable[DAG | SerializedDAG]:
- dagbag = DagBag(read_dags_from_db=True) # type: ignore[call-arg]
+ dagbag = DagBag(read_dags_from_db=True) # type: ignore[misc, call-arg]


It seems to be not related change.

jason810496 · 2025-10-17T04:53:26Z

+ assert any(item.get("id") == prod_model.id for item in prod_list)
+
+ cons_list = runs_by_id[DAG1_RUN2_ID].get("consumed_assets", [])
+ assert isinstance(cons_list, list)
+ assert any(item.get("id") == cons_model.id for item in cons_list)


Would it be better to validate what the exact dict looks like for AssetSummary here?
Somehow like
assertconsumed_assets== [ { "id": ..., "name": ..., ...
which might be more readable for the consumed_assets and produced_assets response.

jason810496 · 2025-10-17T04:58:40Z

+ query = select(DagRun).options(
+ selectinload(DagRun.consumed_asset_events).selectinload(AssetEvent.asset),
+ joinedload(DagRun.dag_model),
+ )


It seems we don't need these selectinload or joinedload statement here?

I think we do need the first part selectinload(DagRun.consumed_asset_events) for the serialization to not emit N lazy db queries at serialization time.
The second part I believe can be moved inside the parameter logic because they seem to be the one using it, and there is no need to always do that eager loading if there is no filtering on consuming or producing assets

jason810496 · 2025-10-17T04:59:46Z

+
+ for dr in dag_runs:
+ consumed_list: list[AssetSummary] = []
+ for ev in getattr(dr, "consumed_asset_events", []) or []:


Suggested change
forevingetattr(dr, "consumed_asset_events", [])or []:
forevingetattr(dr, "consumed_asset_events", []):

jason810496 · 2025-10-17T05:00:28Z

+ for ev in getattr(dr, "consumed_asset_events", []) or []:
+ consumed_list.append(AssetSummary(id=ev.asset_id, name=ev.name, uri=ev.uri, group=ev.group))
+ dr.consumed_assets = consumed_list
+ produced_list = produced_map.get((dr.dag_id, dr.run_id)) or []


Suggested change
produced_list=produced_map.get((dr.dag_id, dr.run_id)) or []
produced_list=produced_map.get((dr.dag_id, dr.run_id), [])

pierrejeambrun

Nice, a few small things to address on top of jason comments and we should be good to move forward.

pierrejeambrun · 2025-10-24T09:24:19Z

+ query = select(DagRun).options(
+ selectinload(DagRun.consumed_asset_events).selectinload(AssetEvent.asset),
+ joinedload(DagRun.dag_model),
+ )


I think we do need the first part selectinload(DagRun.consumed_asset_events) for the serialization to not emit N lazy db queries at serialization time.
The second part I believe can be moved inside the parameter logic because they seem to be the one using it, and there is no need to always do that eager loading if there is no filtering on consuming or producing assets

pierrejeambrun · 2025-10-24T09:30:30Z

+ dag_runs = list(session.scalars(dag_runs_select))
+
+ source_keys = {(dr.dag_id, dr.run_id) for dr in dag_runs}
+ produced_map: dict[tuple[str, str], list[AssetSummary]] = {}


you can use a default dict, that will avoid the 'setdefault' call for each item below.

pierrejeambrun · 2025-10-24T09:34:48Z

+ return select.where(exists_clause)
+
+
+QueryDagRunConsumingAssetFilter = Annotated[


Similarly to other 'search' operating on ilike Pattern suffix should be in the name, take a look at other search params in the same file.

github-actions · 2025-12-20T00:18:41Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

valentinDruzhinin requested review from XD-DENG, ashb, bbovenzi, bugraoz93, ephraimbuddy, jason810496, jscheffl, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners September 24, 2025 03:55

boring-cyborgBot added area:API Airflow's REST/HTTP API area:db-migrations PRs with DB migration area:translations area:UI Related to UI/UX. For Frontend Developers. kind:documentation translation:default labels Sep 24, 2025

valentinDruzhinin force-pushed the feature/53046-dag-view-runs-filters branch from 9006fb6 to d301012Compare September 24, 2025 04:00

vincbeck added the allow translation change This label should be set if we want to bypass translation freeze and change english translations. label Sep 24, 2025

valentinDruzhinin force-pushed the feature/53046-dag-view-runs-filters branch from d301012 to 00679e4Compare September 24, 2025 14:51

bbovenzi reviewed Sep 24, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Outdated

bbovenzi reviewed Sep 24, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py

pierrejeambrun reviewed Sep 24, 2025
View reviewed changes

valentinDruzhinin force-pushed the feature/53046-dag-view-runs-filters branch from 00679e4 to 85b6435Compare September 25, 2025 02:26

valentinDruzhinin requested a review from vincbeck as a code owner September 25, 2025 02:26

pierrejeambrun closed this Sep 30, 2025

pierrejeambrun reopened this Oct 1, 2025

pierrejeambrun changed the title ~~Add DAG view runs filters~~API Add consuming and producing asset filters on list DagRunsOct 1, 2025

valentinDruzhinin force-pushed the feature/53046-dag-view-runs-filters branch 8 times, most recently from 7c98c55 to baef3dcCompare October 3, 2025 20:07

bugraoz93 mentioned this pull request Oct 8, 2025
Airflowctl datamodel generator not updating existed models #56479
Closed
2 tasks

bugraoz93 approved these changes Oct 8, 2025
View reviewed changes

bbovenzi added this to the Airflow 3.2.0 milestone Oct 9, 2025

bbovenzi removed the allow translation change This label should be set if we want to bypass translation freeze and change english translations. label Oct 9, 2025

Add DAG view runs filters
d19356e

valentinDruzhinin force-pushed the feature/53046-dag-view-runs-filters branch from baef3dc to d19356eCompare October 10, 2025 20:05

jason810496 reviewed Oct 17, 2025
View reviewed changes

pierrejeambrun reviewed Oct 24, 2025
View reviewed changes

github-actionsBot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 20, 2025

github-actionsBot closed this Jan 17, 2026

	forevingetattr(dr, "consumed_asset_events", [])or []:
	forevingetattr(dr, "consumed_asset_events", []):

	produced_list=produced_map.get((dr.dag_id, dr.run_id)) or []
	produced_list=produced_map.get((dr.dag_id, dr.run_id), [])

		return select.where(exists_clause)


		QueryDagRunConsumingAssetFilter = Annotated[

		)


		class AssetSummary(PydanticBaseModel):

Uh oh!

Conversation

valentinDruzhinin commented Sep 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RoyLee1224 commented Sep 24, 2025

Uh oh!

valentinDruzhinin commented Sep 24, 2025

Uh oh!

bbovenzi commented Sep 24, 2025

Uh oh!

Uh oh!

valentinDruzhinin commented Sep 24, 2025

Uh oh!

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RoyLee1224 commented Sep 24, 2025

Uh oh!

valentinDruzhinin commented Sep 30, 2025

Uh oh!

pierrejeambrun commented Oct 1, 2025

Uh oh!

valentinDruzhinin commented Oct 3, 2025

Uh oh!

bugraoz93 left a comment

Choose a reason for hiding this comment

Uh oh!

valentinDruzhinin commented Oct 8, 2025

Uh oh!

bbovenzi commented Oct 9, 2025

Uh oh!

bbovenzi commented Oct 9, 2025

Uh oh!

valentinDruzhinin commented Oct 10, 2025

Uh oh!

valentinDruzhinin commented Oct 15, 2025

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

valentinDruzhinin commented Sep 24, 2025•
edited
Loading

pierrejeambrun left a comment •
edited
Loading