Add endpoint to watch dag run until finish by uranusjr · Pull Request #51920 · apache/airflow

uranusjr · 2025-06-19T08:01:40Z

I initially wanted to just enhance the trigger endpoint to optionally stream until the run finishes, but it seems that FastAPI does not like this optionally stream idea. You can do it of course, but would loose a lot of the auto annotation reflection feature. So I opted to have a separate streaming endpoint instead.

This endpoint repeatedly emits a JSON object at the specified interval, until the dag reaches a finished state.

Tests to come.

potiuk · 2025-06-19T10:32:29Z

FYI. Not sure if it helps but OpenAPI 3.2.0 will have support for application/json-seqOAI/OpenAPI-Specification#3730 (planned for August)

uranusjr · 2025-06-25T05:25:14Z

I added some tests for the endpoint, but couldn’t figure out how to test the looping part. Hopefully this is good enough…

pierrejeambrun

Nice, just a few suggestions / nits.

Indeed an additional test case for running / success state would be great.

pierrejeambrun

LGTM, thanks.

Just a few nits, but nothing blocking

kaxil

Some simple docs would be good. Fine as a follow-up too as long as we have a GH issue to not forget.

@vikramkoka could review that too

uranusjr · 2025-07-09T06:34:05Z

Tracking doc addition #53067

(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1) ## New Features: - Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614)) - Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563)) - Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882)) - Implement pattern search for event log endpoint ([#55114](apache/airflow#55114)) - Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263)) - Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302)) - Add `try_number` as filter to task instances ([#54695](apache/airflow#54695)) - Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049)) - Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004)) - Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652)) - Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408)) - Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177)) - Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437)) - Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738)) - Expose DAG parsing duration in the API ([#54752](apache/airflow#54752)) ## New API Endpoints: - Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776)) - Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920)) - Add TI bulk actions endpoint ([#50443](apache/airflow#50443)) - Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657)) ## Deprecations: - Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150)) ## Bug Fixes: - Fix dag import error modal pagination ([#55719](apache/airflow#55719))

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

boring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Jun 19, 2025

uranusjr mentioned this pull request Jun 19, 2025
Enable Inference Execution / Synchronous DAG Execution #51711
Closed

uranusjr added this to the Airflow 3.1.0 milestone Jun 24, 2025

uranusjr force-pushed the stream-until-run-complete branch 2 times, most recently from 231597b to 007a051Compare June 25, 2025 04:53

uranusjr marked this pull request as ready for review June 25, 2025 05:25

uranusjr requested review from bbovenzi, bugraoz93, ephraimbuddy, jason810496, jscheffl, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners June 25, 2025 05:25

pierrejeambrun reviewed Jun 25, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_run.py Outdated
Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
Comment threadairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py

jason810496 reviewed Jun 27, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py

uranusjr force-pushed the stream-until-run-complete branch from 007a051 to 152bd6dCompare July 3, 2025 09:31

pierrejeambrun approved these changes Jul 3, 2025
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py Outdated
Comment threadairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py Outdated
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml

uranusjr force-pushed the stream-until-run-complete branch from 152bd6d to 34badc1Compare July 7, 2025 07:04

pierrejeambrun approved these changes Jul 7, 2025
View reviewed changes

uranusjr force-pushed the stream-until-run-complete branch 2 times, most recently from 96812e6 to 422cd94Compare July 7, 2025 19:27

bugraoz93 approved these changes Jul 7, 2025
View reviewed changes

kaxil reviewed Jul 7, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Outdated

kaxil approved these changes Jul 7, 2025
View reviewed changes

Add endpoint to watch dag run until finish
e1e3149

uranusjr added 4 commits July 9, 2025 07:40

Include xcom in result if requested
edf8787

Use 'params' instead of literal params
77ec3ac

Mark endpoint as experimental
f852f6b

Rename param key
58d7fa5

uranusjr force-pushed the stream-until-run-complete branch from 422cd94 to 58d7fa5Compare July 9, 2025 05:40

uranusjr mentioned this pull request Jul 9, 2025
Add docs for the wait dagrun endpoint #53067
Closed
1 task

uranusjr merged commit 5fbbf64 into apache:mainJul 9, 2025

uranusjr deleted the stream-until-run-complete branch July 9, 2025 06:34

HsiuChuanHsu pushed a commit to HsiuChuanHsu/airflow that referenced this pull request Jul 10, 2025
Add endpoint to watch dag run until finish (apache#51920)
2a6e549

stephen-bracken pushed a commit to stephen-bracken/airflow that referenced this pull request Jul 15, 2025
Add endpoint to watch dag run until finish (apache#51920)
ad1932e

kaxil mentioned this pull request Oct 22, 2025
Update Python Client to 3.1.0 apache/airflow-client-python#138
Merged

kaxil mentioned this pull request Apr 15, 2026
Fix dispose_orm() not disposing async engine on shutdown #65274
Merged

kaxil mentioned this pull request Apr 15, 2026
Fix async engine missing pool_recycle and pool_pre_ping configuration #65276
Merged

vatsrahul1001 mentioned this pull request Apr 15, 2026
[v3-2-test] Fix dispose_orm() not disposing async engine on shutdown (#65274) #65284
Merged

vatsrahul1001 mentioned this pull request Apr 15, 2026
Status of testing of Apache Airflow 3.2.1rc3 #65336
Closed
53 tasks

Navigation Menu

Uh oh!

Add endpoint to watch dag run until finish - #51920

Add endpoint to watch dag run until finish#51920
uranusjr merged 5 commits into
apache:mainfrom
astronomer:stream-until-run-complete

uranusjr commented Jun 19, 2025•
edited
Loading

Uh oh!

potiuk commented Jun 19, 2025

Uh oh!

uranusjr commented Jun 25, 2025

Uh oh!

pierrejeambrun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil left a comment

Uh oh!

uranusjr commented Jul 9, 2025•
edited
Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

uranusjr commented Jun 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Jun 19, 2025

Uh oh!

uranusjr commented Jun 25, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

uranusjr commented Jul 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

uranusjr commented Jun 19, 2025•
edited
Loading

uranusjr commented Jul 9, 2025•
edited
Loading