Skip to content

Backport fix to allow pickling of Loggers to Python 3.6 - #18798

Merged
ashb merged 1 commit into
apache:mainfrom
astronomer:logger-pickle-fix-py3.6
Oct 7, 2021
Merged

Backport fix to allow pickling of Loggers to Python 3.6#18798
ashb merged 1 commit into
apache:mainfrom
astronomer:logger-pickle-fix-py3.6

Conversation

@ashb

@ashbashb commented Oct 7, 2021

Copy link
Copy Markdown
Member

When sending objects around via multiprocessing (on Python 3.6 or lower)
it would fail if that object contained a Logger object.

To fix that we have "backported" the change in Python 3.7 to make Logger
objects be pickled "by name". (In Python 3.7 the change adds
__reduce__ methods on to the Logger and RootLogger objects, but here
we achieve it copyreg stdlib module so we don't monkeypatch
anything.)

This mainly applies to using Kubernetes client >12 (which is not
currently possible as we restrict that version) but this adds support
for it anywhere it might happen inside Python 3.6.

When sending objects around via multiprocessing (on Python 3.6 or lower)
it would fail if that object contained a Logger object.
To fix that we have "backported" the change in Python 3.7 to make Logger
objects be pickled "by name". (In Python 3.7 the change adds
`__reduce__` methods on to the Logger and RootLogger objects, but here
we achieve it `copyreg` stdlib module so we don't monkeypatch
anything.)
This mainly applies to using Kubernetes client >12 (which is not
currently possible as we restrict that version) but this adds support
for it anywhere it might happen inside Python 3.6.
@ashbashb added the full tests needed We need to run full set of tests for this PR to merge label Oct 7, 2021
@ashb
ashb requested review from kaxil and potiukOctober 7, 2021 09:43
@ashbashb added this to the Airflow 2.2.0 milestone Oct 7, 2021

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one!

@ashb
ashb merged commit c10442e into apache:mainOct 7, 2021
@ashb
ashb deleted the logger-pickle-fix-py3.6 branch October 7, 2021 13:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests neededWe need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ashb@potiuk@kaxil