Skip to content

AIP-81 Add Insert Multiple Pools API - #44121

Merged
pierrejeambrun merged 9 commits into
apache:mainfrom
jason810496:feature/AIP-81/add-bulk-insert-pools-api
Nov 22, 2024
Merged

AIP-81 Add Insert Multiple Pools API#44121
pierrejeambrun merged 9 commits into
apache:mainfrom
jason810496:feature/AIP-81/add-bulk-insert-pools-api

Conversation

@jason810496

@jason810496jason810496 commented Nov 18, 2024

Copy link
Copy Markdown
Member

closes: #43896
related: #43657

Fix: Add max_length=256 for PoolPostBody.pool

The model defined in https://github.com/apache/airflow/blob/main/airflow/models/pool.py#L54 enforces a string length constraint on Pool.pool. To maintain consistency, this constraint should also be validated at the router level.

Refactor: Handle Duplicate Cases in Insert Pool

The Insert Pool (single insert) functionality should account for cases where a duplicate pool name is provided, as Pool.pool is defined with a unique constraint.

Feat: Add Insert Multiple Pools API

A new API endpoint for inserting multiple pools has been introduced as /pools/bulk. Alternative names such as /pools/batch or /pools/multiple were considered. Feedback on which name best describes the endpoint is welcome.

@boring-cyborgboring-cyborgBot added the area:UI Related to UI/UX. For Frontend Developers. label Nov 18, 2024
rawwar
rawwar reviewed Nov 18, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/datamodels/pools.py
rawwar
rawwar reviewed Nov 18, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
rawwar
rawwar reviewed Nov 18, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/pools.py
rawwar
rawwar reviewed Nov 18, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
rawwar
rawwar reviewed Nov 18, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
pierrejeambrun
pierrejeambrun reviewed Nov 18, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks

Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
Comment threadairflow/api_fastapi/core_api/datamodels/pools.py Outdated
@jason810496
jason810496force-pushed the feature/AIP-81/add-bulk-insert-pools-api branch from 9fa560e to 42e8a51CompareNovember 18, 2024 12:53
@bugraoz93bugraoz93 added the area:API Airflow's REST/HTTP API label Nov 19, 2024
@bugraoz93

bugraoz93 commented Nov 19, 2024

Copy link
Copy Markdown
Contributor

Looks good! Thanks for the changes @jason810496! I believe the CI failure is likely a transient error caused by a DNS issue that couldn’t resolve AWS S3 at that moment.

@Lee-W

Lee-W commented Nov 20, 2024

Copy link
Copy Markdown
Member

Looks good! Thanks for the changes @jason810496! I believe the CI failure is likely a transient error caused by a DNS issue that couldn’t resolve AWS S3 at that moment.

just reran. seems to work fine 👀

@jason810496
jason810496force-pushed the feature/AIP-81/add-bulk-insert-pools-api branch from 42e8a51 to 1ec682fCompareNovember 20, 2024 12:20
@jason810496

jason810496 commented Nov 20, 2024

Copy link
Copy Markdown
MemberAuthor

Just resolved the issue with inserting duplicate pools by handling the database exception rather than adding extra logic to the datamodels or performing additional database queries.
By the way, I’m happy to work further on refactoring to handle duplicate insertion cases across all endpoints 🙌

pierrejeambrun
pierrejeambrun reviewed Nov 20, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, one minor nit, we can merge then.

Comment threadtests/api_fastapi/core_api/routes/public/test_pools.py Outdated
pierrejeambrun
pierrejeambrun previously approved these changes Nov 20, 2024
View reviewed changes
@pierrejeambrun

pierrejeambrun commented Nov 20, 2024

Copy link
Copy Markdown
Member

By the way, I’m happy to work further on refactoring to handle duplicate insertion cases across all endpoints 🙌

That would be great. You can create an issue to track that if you're not going to work on it directly just so we do not forget about it. If you plan on tackling that right away, no need to create an issue opening a PR is perfectly fine.

pierrejeambrun
pierrejeambrun reviewed Nov 20, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this is not the best approach

Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
@jason810496
jason810496force-pushed the feature/AIP-81/add-bulk-insert-pools-api branch from bde6004 to 510e373CompareNovember 21, 2024 07:47
pierrejeambrun
pierrejeambrun approved these changes Nov 21, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice. I like the exception handler class approach (private classes, only expose array of handlers), and the factory pattern for registering the handlers, good idea.

A couple of nitpicks, ready to merge after that.

Comment threadairflow/api_fastapi/common/exceptions.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/pools.py Outdated
@jason810496
jason810496force-pushed the feature/AIP-81/add-bulk-insert-pools-api branch from 510e373 to c9b67c1CompareNovember 22, 2024 07:59
@jason810496

jason810496 commented Nov 22, 2024

Copy link
Copy Markdown
MemberAuthor

Just fixed the nit. I will refactor all endpoints to adopt the global error handler for managing unique constraint violation errors in further PR.

@pierrejeambrun

pierrejeambrun commented Nov 22, 2024

Copy link
Copy Markdown
Member

Just fixed the nit. I will refactor all endpoints to adopt the global error handler for managing unique constraint violation errors in further PR.

Nice. 🎉

What is great about that is the exception_handler is global to the application. All endpoints should benefit from it without any additional specifications. The only thing I can thing of is to correctly document the 409 that can be possibly returned. (That's the benefit of this approach)

@pierrejeambrun
pierrejeambrun merged commit f33166a into apache:mainNov 22, 2024
@jason810496jason810496 mentioned this pull request Nov 24, 2024
Merged
@jedcunninghamjedcunningham changed the title AIP-81 Add Insert Multiple Pools APIAIP-84 Add Insert Multiple Pools APIDec 18, 2024
@jedcunninghamjedcunningham changed the title AIP-84 Add Insert Multiple Pools APIAIP-81 Add Insert Multiple Pools APIDec 18, 2024
@jedcunninghamjedcunningham added the AIP-81 Enhanced Security in CLI via Integration of API label Dec 18, 2024
@pierrejeambrunpierrejeambrun added the AIP-84 Modern Rest API label Jan 23, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
@jason810496@got686-yandex
AIP-81 Add Insert Multiple Pools API (apache#44121)
72d8f73
* Add bulk post pools, refactor post pool
* Add 409 case for TestPostPool
* Add test for bulk post pools
* Remove unused status code, rename post_body to body
* Refactor duplicate pool insert handling
- handle exception from db level instead of application level
* Add global database exception handler for fastapi
* Remove manual handle for unique constraint exc
* Refactor test_pools
* Fix bound for TypeVar, type for comment
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-WLee-WLee-W left review comments
@bugraoz93bugraoz93bugraoz93 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@rawwarrawwarrawwar left review comments
@bbovenzibbovenziAwaiting requested review from bbovenzi
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamilton
@jscheffljschefflAwaiting requested review from jscheffl
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy

Assignees

No one assigned

Labels

AIP-81Enhanced Security in CLI via Integration of APIAIP-84Modern Rest APIarea:APIAirflow's REST/HTTP APIarea:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AIP-81 Implement POST/Insert Multiple Pools in FastAPI

6 participants

@jason810496@bugraoz93@Lee-W@pierrejeambrun@rawwar@jedcunningham