Uh oh!
There was an error while loading. Please reload this page.
Turn Pydantic into an optional dependency - #37320
Merged
Merged
Conversation
potiuk
requested review from
Taragolis, eladkal, ephraimbuddy, hussein-awala, jedcunningham, mhenc and vincbeckFebruary 10, 2024 22:52
potiukforce-pushed
the
make-pydantic-optional-dependency
branch
4 times, most recently
from
February 11, 2024 01:09
312f55d to
ac41af9Comparepotiukforce-pushed
the
make-pydantic-optional-dependency
branch
from
February 11, 2024 01:54
ac41af9 to
676a4ecComparepotiuk
commented
Feb 11, 2024
Uh oh!
There was an error while loading. Please reload this page.
potiukforce-pushed
the
make-pydantic-optional-dependency
branch
from
February 11, 2024 01:57
676a4ec to
88c435aComparepotiuk
commented
Feb 11, 2024
Uh oh!
There was an error while loading. Please reload this page.
potiukforce-pushed
the
make-pydantic-optional-dependency
branch
3 times, most recently
from
February 11, 2024 21:10
0f41e34 to
b9b06a1Comparepotiuk
commented
Feb 12, 2024
Uh oh!
There was an error while loading. Please reload this page.
potiuk
commented
Feb 12, 2024
Uh oh!
There was an error while loading. Please reload this page.
potiuk
commented
Feb 12, 2024
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db.
potiukforce-pushed
the
make-pydantic-optional-dependency
branch
from
February 12, 2024 20:42
3d6128a to
16aeb19Comparepotiuk added a commit
that referenced
this pull request
Feb 13, 2024
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db. (cherry picked from commit c3f48ee)
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Feb 13, 2024
The changes in apache#37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests.
potiuk added a commit
that referenced
this pull request
Feb 13, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests.
This was referenced Feb 16, 2024
ephraimbuddy pushed a commit
that referenced
this pull request
Feb 20, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests. (cherry picked from commit cbc9af0)
ephraimbuddy pushed a commit
that referenced
this pull request
Feb 22, 2024
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db. (cherry picked from commit c3f48ee)
ephraimbuddy pushed a commit
that referenced
this pull request
Feb 22, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests. (cherry picked from commit cbc9af0)
68 tasks
79 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed.
It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found.
This should enable less conflicts when installing 3rd-party libraries with Airflow.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.