Skip to content

AIP-84 Migrate Clear Dag Run public endpoint to FastAPI - #42975

Merged
pierrejeambrun merged 24 commits into
apache:mainfrom
rawwar:kalyan/API-84/clear_dag_run
Nov 15, 2024
Merged

AIP-84 Migrate Clear Dag Run public endpoint to FastAPI #42975
pierrejeambrun merged 24 commits into
apache:mainfrom
rawwar:kalyan/API-84/clear_dag_run

Conversation

@rawwar

Copy link
Copy Markdown
Contributor

related to #42701

@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Oct 13, 2024
@pierrejeambrunpierrejeambrun added the legacy api Whether legacy API changes should be allowed in PR label Oct 14, 2024
@rawwarrawwar closed this Oct 30, 2024
@rawwar
rawwarforce-pushed the kalyan/API-84/clear_dag_run branch from a418bcf to 4fc16f1CompareOctober 30, 2024 06:39
@rawwarrawwar reopened this Oct 30, 2024
@rawwar

rawwar commented Nov 3, 2024

Copy link
Copy Markdown
ContributorAuthor

@pierrejeambrun , I noticed that in legacy implementation, for clear dag run endpoint with dry_run=True, the response is supposed to be a TaskInstanceCollection. However, the response only includes few attributes of TI. Below is an example response :

{
"task_instances": [
{
"dag_id": "example_astronauts",
"dag_run_id": "manual__2024-11-03T14:05:08.832062+00:00",
"execution_date": "2024-11-03T14:05:08.832062+00:00",
"task_id": "print_astronaut_craft"
},{
"dag_id": "example_astronauts",
"dag_run_id": "manual__2024-11-03T14:05:08.832062+00:00",
"execution_date": "2024-11-03T14:05:08.832062+00:00",
"task_id": "get_astronauts"
}

Related schema: TaskInstanceReferenceSchema
Should I keep the same? Or should I return all the details?

I tried to return all details. But, to do that, it seems I can't reuse the methods in the legacy implementation as it is causing the following error.

Error extracting attribute: DetachedInstanceError: Parent instance <TaskInstance at 0x31af2e880> is not bound to a Session; lazy load operation of attribute 'task_instance_note' cannot proceed (Background on this error at: https://sqlalche.me/e/14/bhk3) [type=get_attribute_error, input_value=<TaskInstance: example_as....832062+00:00 [success]>, input_type=TaskInstance]

I guess, I just need to use joinload. But, thinking if I should update here or just rewrite a select query within clear_dag_run method

@pierrejeambrun

pierrejeambrun commented Nov 5, 2024

Copy link
Copy Markdown
Member

I think we can do the same and return a partial response. There is a way to do that in fastapi specifying the response model. And we need to document it in the swagger with example responses.

DetachedInstance error is most certainly due to a bad session handling. (session used to fetch objects is closed too early or something similar)

@rawwar
rawwar marked this pull request as ready for review November 8, 2024 03:09
Comment threadairflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
@rawwar

Copy link
Copy Markdown
ContributorAuthor

I think we can do the same and return a partial response. There is a way to do that in fastapi specifying the response model. And we need to document it in the swagger with example responses.

I went with returning the entire object. Is that fine? If not, I can return the partial response. I already looked into how to do it

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

A few suggestions/questions

Comment threadairflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
Comment threadairflow/api_fastapi/core_api/datamodels/dag_run.py Outdated
Comment threadtests/api_fastapi/core_api/routes/public/test_dag_run.py Outdated
rawwarand others added 4 commits November 14, 2024 09:57

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, a few minor suggestions and ready to merge, thanks

Comment threadairflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/dag_run.py Outdated
@pierrejeambrun
pierrejeambrun merged commit c3aabba into apache:mainNov 15, 2024
@rawwar
rawwar deleted the kalyan/API-84/clear_dag_run branch November 15, 2024 08:24
hardeybisey pushed a commit to hardeybisey/airflow that referenced this pull request Nov 15, 2024
* add clear_dag_run
* add tests
* Merge branch 'main' of https://github.com/apache/airflow into kalyan/API-84/clear_dag_run
* add ti response
* add
* use logical_date
* fix tests
* remove async
* Update airflow/api_fastapi/core_api/routes/public/dag_run.py
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* Update airflow/api_fastapi/core_api/datamodels/dag_run.py
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* remove type ignore
* update ti state and assert it
* reuse state
* remove breakpoint
* feedback
---------
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.legacy apiWhether legacy API changes should be allowed in PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@rawwar@pierrejeambrun