Skip to content

Revert "Redirect old location module imports to standard provider (#4… - #43946

Merged
gopidesupavan merged 1 commit into
apache:mainfrom
potiuk:revert-module-redirection
Nov 13, 2024
Merged

Revert "Redirect old location module imports to standard provider (#4…#43946
gopidesupavan merged 1 commit into
apache:mainfrom
potiuk:revert-module-redirection

Conversation

@potiuk

Copy link
Copy Markdown
Member

…3610)"

There is a very interesting breaking change introduced in Python 3.11 that will likely mean that we should not use MetaPathFinder for old standard provider classes redirection.

There was a change introduced in Python 3.11 that caused that module might not be found in some cases (for example when unit test patches the path) when the module is loaded as a different module (i.e. a.b differs from sys.modules['a.b'])

This is tracked in CPython via:
python/cpython#117860

This causes standard operator's tests fail in Python 3.11 and 3.12 when the providers/tests/standard/test_module_redirect_finder.py is executed before - i.e. the standard modules are loaded as old modules.

This reverts commit 5de2e73.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…ache#43610)"
There is a very interesting breaking change introduced in Python 3.11
that will likely mean that we should not use MetaPathFinder for old
standard provider classes redirection.
There was a change introduced in Python 3.11 that caused that
module might not be found in some cases (for example when
unit test patches the path) when the module is loaded as
a different module (i.e. a.b differs from sys.modules['a.b'])
This is tracked in CPython via:
python/cpython#117860
This causes standard operator's tests fail in Python 3.11 and
3.12 when the providers/tests/standard/test_module_redirect_finder.py
is executed before - i.e. the standard modules are loaded as
old modules.
This reverts commit 5de2e73.
@gopidesupavan
gopidesupavan merged commit 807fd6d into apache:mainNov 13, 2024
@gopidesupavan

Copy link
Copy Markdown
Member

really interesting one :)

@potiuk

Copy link
Copy Markdown
MemberAuthor

really interesting one :)

Indeed - that is about third time when our test harness (which I am reeally happy about) uncover a bug in CPython.
For me this one case is a single most important reason why we should have all the matrices and canary builds running everything. Once evey half a year or so, they discover an issue in one or two of the combos that would take us weeks of investigation if it would have happened in "production" - and it happens way, way, way before it is even close to being released.

ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
…ache#43610)" (apache#43946)
There is a very interesting breaking change introduced in Python 3.11
that will likely mean that we should not use MetaPathFinder for old
standard provider classes redirection.
There was a change introduced in Python 3.11 that caused that
module might not be found in some cases (for example when
unit test patches the path) when the module is loaded as
a different module (i.e. a.b differs from sys.modules['a.b'])
This is tracked in CPython via:
python/cpython#117860
This causes standard operator's tests fail in Python 3.11 and
3.12 when the providers/tests/standard/test_module_redirect_finder.py
is executed before - i.e. the standard modules are loaded as
old modules.
This reverts commit 5de2e73.
@potiuk
potiuk deleted the revert-module-redirection branch April 21, 2026 15:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@gopidesupavan