Exposing DAG Parsing duration in the UI by SamWheating · Pull Request #54752 · apache/airflow

SamWheating · 2025-08-20T20:31:31Z

This PR adds a last_parse_time column to the DAG table, and updates it during the DAG processing loop. This field is then exposed in the DAG details API and surfaced to users in the Code view like so:

As well as the DAG details pane, like so:

Which can be used to identify slow-parsing DAGs without having to look in the processor logs or metrics.

In the future we could also expose this data elsewhere, for example adding a "slowest parsing files" table to the cluster overview page, or something similar.

Anyways, I still need to do a bunch of cleanup here, add new test coverage and update a bunch of failing tests, but I wanted to get some early feedback on the implementation here, namely:

Thoughts on exposing this information on the code tab of the DAG details page?
Thoughts on the wording - is last_parse_duration / Parse Duration the most accurate naming here? I'm not even sure if my grammar is correct.

jscheffl

For me this change makes sense and I like it. I would support adding this as a feature and also displaying this on the Code tab!

potiuk · 2025-08-21T14:21:54Z

I like it a lot too. I Think though - not sure if we have it already, it would also make sense to have a way in the UI to see all the Dags that are coming from the same dag file - for example by clicking on the "parse time" or smth. That could give bette insight in case of multiple-dags-in-one-file case.

SamWheating · 2025-08-21T17:40:26Z

I like it a lot too. I Think though - not sure if we have it already, it would also make sense to have a way in the UI to see all the Dags that are coming from the same dag file - for example by clicking on the "parse time" or smth. That could give bette insight in case of multiple-dags-in-one-file case.

This is a good suggestion and should be totally possible since all of this information is already available in the DB, but I have to think about how to best present this in the UI 🤔 It might get messy if we start putting information about other DAGs into the code pane.

Maybe we should make a separate issue for this suggestion? I can propose some options once this one has been discussed a bit

potiuk · 2025-08-22T18:41:43Z

Maybe we should make a separate issue for this suggestion? I can propose some options once this one has been discussed a bit

yep

SamWheating · 2025-09-04T03:05:47Z

Ok, I have fixed all of the merge conflicts - this should be ready for review again.

jason810496

Thanks for the PR! Nice feature! LGTM overall.

It seems alembic related change need to be generated again by airflow/contributing-docs/14_metadata_database_updates - How to Resolve Conflicts When Rebasing

Additionally, the ruff static check needs fix as well, thanks!

prek ruff --all-files && prek run ruff-format --all-files

jscheffl

For (still) this looks good, would be be accepted to have this in 3.1 even though new translation strings are added?

potiuk · 2025-09-06T15:57:59Z

For (still) this looks good, would be be accepted to have this in 3.1 even though new translation strings are added?

I rebased it to follow the new process. This might be first one to follow it (cc: @shahar1 @jscheffl ) - I am happy with getting that one in after applying the allow translation change. Whoever merges it in, should notify everyone on #i18n slack channel. :)

potiuk · 2025-09-06T15:58:41Z

Closed/reopened to re-run with the label applied.

jason810496 · 2025-09-06T16:54:18Z

Whoever merges it in, should notify everyone on #i18n slack channel. :)

All CI checks have passed. Let's see if it works as expected! 👀

Update:

Got "This branch is out-of-date with the base branch" so I can't merge.
Just clicked "Update with rebase" to resolve it.

jscheffl

I'd be OK to merge before 3.1 and having this in, even if two translation keys added.
I am not sure if a kind of "LAZY CONSENSUS" applies here but I'd be OK to merge tomorrow EOB CEST if nobody objects.

* store DAG parsing duration and display it in UI * static fixes, regenerate ERD

(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1) ## New Features: - Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614)) - Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563)) - Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882)) - Implement pattern search for event log endpoint ([#55114](apache/airflow#55114)) - Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263)) - Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302)) - Add `try_number` as filter to task instances ([#54695](apache/airflow#54695)) - Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049)) - Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004)) - Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652)) - Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408)) - Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177)) - Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437)) - Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738)) - Expose DAG parsing duration in the API ([#54752](apache/airflow#54752)) ## New API Endpoints: - Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776)) - Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920)) - Add TI bulk actions endpoint ([#50443](apache/airflow#50443)) - Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657)) ## Deprecations: - Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150)) ## Bug Fixes: - Fix dag import error modal pagination ([#55719](apache/airflow#55719))

Cir02 · 2025-11-17T14:52:04Z

Hi, is this feature coming to the rest of versions? Airflow 2.11 or similar? thanks

jedcunningham · 2025-11-17T15:00:42Z

No, Airflow 2 is only receiving bugfixes now, not improvements.

potiuk · 2025-11-17T17:15:05Z

And only critical ones.

SamWheating requested review from XD-DENG, ashb, bbovenzi, bugraoz93, ephraimbuddy, jason810496, jedcunningham, jscheffl, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners August 20, 2025 20:31

boring-cyborgBot added area:API Airflow's REST/HTTP API area:DAG-processing area:db-migrations PRs with DB migration area:translations area:UI Related to UI/UX. For Frontend Developers. kind:documentation translation:default labels Aug 20, 2025

jscheffl reviewed Aug 20, 2025
View reviewed changes

SamWheating force-pushed the sw-add-last-parse-duration-to-ui branch 2 times, most recently from bc2c3dc to 180ab30Compare August 20, 2025 21:35

SamWheating requested review from kaxil and potiuk as code owners August 20, 2025 22:31

SamWheating force-pushed the sw-add-last-parse-duration-to-ui branch from a569d8e to 4ce72a8Compare August 20, 2025 23:37

SamWheating commented Aug 21, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/models/dagbag.py

SamWheating changed the title ~~(WIP) Exposing DAG Parsing duration in the UI~~Exposing DAG Parsing duration in the UIAug 21, 2025

SamWheating force-pushed the sw-add-last-parse-duration-to-ui branch from a2b53a4 to 256f0a8Compare September 4, 2025 03:03

SamWheating requested a review from bolkedebruin as a code owner September 4, 2025 03:03

jason810496 reviewed Sep 4, 2025
View reviewed changes

jscheffl approved these changes Sep 4, 2025
View reviewed changes

potiuk force-pushed the sw-add-last-parse-duration-to-ui branch from 565bd34 to ae3dd29Compare September 6, 2025 15:55

potiuk added the allow translation change This label should be set if we want to bypass translation freeze and change english translations. label Sep 6, 2025

potiuk approved these changes Sep 6, 2025
View reviewed changes

potiuk closed this Sep 6, 2025

potiuk reopened this Sep 6, 2025

SamWheating added 2 commits September 7, 2025 00:56

store DAG parsing duration and display it in UI
521c2cc

static fixes, regenerate ERD
a16bb93

jason810496 force-pushed the sw-add-last-parse-duration-to-ui branch from ae3dd29 to a16bb93Compare September 6, 2025 16:56

jscheffl approved these changes Sep 6, 2025
View reviewed changes

jscheffl merged commit 5593995 into apache:mainSep 7, 2025
108 checks passed

RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Sep 8, 2025
Exposing DAG Parsing duration in the UI (apache#54752)
bca0443
* store DAG parsing duration and display it in UI * static fixes, regenerate ERD

kaxil mentioned this pull request Oct 22, 2025
Update Python Client to 3.1.0 apache/airflow-client-python#138
Merged

Uh oh!

Conversation

SamWheating commented Aug 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

Uh oh!

SamWheating commented Aug 21, 2025

Uh oh!

potiuk commented Aug 22, 2025

Uh oh!

SamWheating commented Sep 4, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Sep 6, 2025

Uh oh!

potiuk commented Sep 6, 2025

Uh oh!

jason810496 commented Sep 6, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Cir02 commented Nov 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jedcunningham commented Nov 17, 2025

Uh oh!

potiuk commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

SamWheating commented Aug 20, 2025•
edited
Loading

SamWheating commented Sep 4, 2025•
edited
Loading

jason810496 commented Sep 6, 2025•
edited
Loading

Cir02 commented Nov 17, 2025•
edited
Loading