Skip to content

Make core compatible with pendulum 3 - #34744

Closed
Taragolis wants to merge 3 commits into
apache:mainfrom
Taragolis:core-compat-with-pendulum-3
Closed

Make core compatible with pendulum 3#34744
Taragolis wants to merge 3 commits into
apache:mainfrom
Taragolis:core-compat-with-pendulum-3

Conversation

@Taragolis

Copy link
Copy Markdown
Contributor

In pendulum 3 (beta) there is no access to pendulum.tz.timezone anymore otherwise we should call pendulum.timezone for convert string/integer to pendulum timezones

>>>importpendulum>>>pendulum.__version__'2.1.2'>>>pendulum.tz.timezone("UTC")
Timezone('UTC')
>>>pendulum.tz.timezone("Europe/London")
Timezone('Europe/London')
>>>pendulum.tz.timezone(3600)
Timezone('+01:00')
>>>pendulum.timezone("UTC")
Timezone('UTC')
>>>pendulum.timezone("Europe/London")
Timezone('Europe/London')
>>>pendulum.timezone(3600)
Timezone('+01:00')
>>>importpendulum>>>pendulum.__version__'3.0.0b1'>>>pendulum.tz.timezone("UTC")
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>TypeError: 'module'objectisnotcallable>>>pendulum.tz.timezone("Europe/London")
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>TypeError: 'module'objectisnotcallable>>>pendulum.timezone("UTC")
Timezone('UTC')
>>>pendulum.timezone("Europe/London")
Timezone('Europe/London')
>>>pendulum.timezone(3600)
FixedTimezone(3600, name="+01:00")

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Comment threadairflow/utils/timezone.py Outdated
Comment threadtests/models/test_dag.py Outdated
Comment threadairflow/utils/timezone.py Outdated
@TaragolisTaragolis added the full tests needed We need to run full set of tests for this PR to merge label Oct 4, 2023
@Taragolis
Taragolisforce-pushed the core-compat-with-pendulum-3 branch from 4b3d5f8 to 2fc1d4eCompareOctober 4, 2023 11:00
@TaragolisTaragolis changed the title Make core compatible with pendulum 3Make core compatible with pendulum 3Oct 4, 2023
Comment threadairflow/serialization/serialized_objects.py Outdated
Comment threadairflow/utils/timezone.py Outdated

@uranusjruranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally lgtm, just a couple of nits

@bolkedebruinbolkedebruin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions and small nits. Generally looks fine, thanks for putting in the work.

Comment threadairflow/utils/timezone.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there an issue moving to Pendulum 3 UTC? So moving from FixedTimezone to Timezone? FixedTimezone derives from Timezone.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is interesting question. In general speaking right now we use two different version of UTC timezone in airflow codebase.

  1. FixedTimezone created by pendulum.tz.timezone("UTC") in most part of codebase
  2. Timezone created by pendulum.tz.timezone.Timezone("UTC") in timetables

This is a two different objects with different attributes however it should work exactly the same. So it might be a good idea to try to resolve different implementation of UTC timezone in airflow codebase, I just choose to keep the same as it now in most parts, but I happy to change it to pendulum.tz.timezone.Timezone("UTC")

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bolkedebruin
Well as we could see on our CI Timezone("UTC") in pendulum 2.x might not work into some specific cases such

else:
holidays=holiday_calendar.holidays(start=next_start, end=next_start).to_pydatetime()

This code works fine in pendulum 3, however it failed (or spawn a lot of warnings) in 2.1.2

frompendulum.tz.timezoneimportFixedTimezone, Timezonefromdatetimeimportdatetimefrompandas.tseries.holidayimportUSFederalHolidayCalendarholiday_calendar=USFederalHolidayCalendar()
forutcin [FixedTimezone(offset=0, name="UTC"), Timezone("UTC")]:
print(f" {utc}{type(utc).__name__} ".center(72, "="))
next_start=datetime(2021, 9, 5, tzinfo=utc)
result=holiday_calendar.holidays(start=next_start, end=next_start)
print(f"Result: {next_start}")

@TaragolisTaragolisNov 22, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And revert it back to FixedTimezone

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and what about using

fromdatetimeimporttimezoneutc=timezone.utc

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check this one, some of the methods pendulum 2 have a problem with non-pendulum's timezones, e.g. ZoneInfo

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then leave it for now, we can switch later

Comment threadairflow/utils/timezone.py Outdated
Comment threadtests/models/test_dag.py Outdated
potiuk added a commit to potiuk/airflow that referenced this pull request Oct 23, 2023
Python 3.12 has a few breaking changes comparing to earlier versions.
While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the
first one for a long time that started to break things more
aggressively.
For now we know that Airflow will not work with Python 3.12 mainly
because of distutils removal (https://peps.python.org/pep-0632/) and
not because of Airflow's usage of it but pendulum's before version 3.
While we are working on getting Pendulum 3 support in apache#34744 and
the apache#34746, there are likely other dependencies that have similar
issue.
Until we fix it and add official 3.12 support, we can limit airflow
to not be installable on 3.12.
@Taragolis
Taragolisforce-pushed the core-compat-with-pendulum-3 branch from 2fc1d4e to e4232e9CompareOctober 23, 2023 20:00
potiuk added a commit that referenced this pull request Oct 25, 2023
Python 3.12 has a few breaking changes comparing to earlier versions.
While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the
first one for a long time that started to break things more
aggressively.
For now we know that Airflow will not work with Python 3.12 mainly
because of distutils removal (https://peps.python.org/pep-0632/) and
not because of Airflow's usage of it but pendulum's before version 3.
While we are working on getting Pendulum 3 support in #34744 and
the #34746, there are likely other dependencies that have similar
issue.
Until we fix it and add official 3.12 support, we can limit airflow
to not be installable on 3.12.
potiuk added a commit that referenced this pull request Oct 29, 2023
Python 3.12 has a few breaking changes comparing to earlier versions.
While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the
first one for a long time that started to break things more
aggressively.
For now we know that Airflow will not work with Python 3.12 mainly
because of distutils removal (https://peps.python.org/pep-0632/) and
not because of Airflow's usage of it but pendulum's before version 3.
While we are working on getting Pendulum 3 support in #34744 and
the #34746, there are likely other dependencies that have similar
issue.
Until we fix it and add official 3.12 support, we can limit airflow
to not be installable on 3.12.
(cherry picked from commit 020691f)
@Felix-neko

Felix-neko commented Nov 6, 2023

Copy link
Copy Markdown

Thank you for your PR, bro!
I've checked this version with #35483 DAG and the the problem with NameError: name 'timedelta' is not defined vanished.

Taragolisand others added 3 commits November 22, 2023 12:45
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Co-authored-by: bolkedebruin <bolkedebruin@users.noreply.github.com>
@Taragolis
Taragolisforce-pushed the core-compat-with-pendulum-3 branch from e4232e9 to ee27310CompareNovember 22, 2023 08:53
@Taragolis

Copy link
Copy Markdown
ContributorAuthor

@bolkedebruin Would you have a look again? Do you have any concern?

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bolkedebruin

Copy link
Copy Markdown
Contributor

LGTM with one small question see above.

@Taragolis

Copy link
Copy Markdown
ContributorAuthor

I've cherry picked changes into the #35798 and it fail in multiple new places, so I would suggest to close this one, and And continue working in that PR so we would sure that changes also would potentially work in pendulum 3

@Taragolis

Copy link
Copy Markdown
ContributorAuthor

closed in favor of #35798

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:serializationarea:Triggererfull tests neededWe need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

@Taragolis@Felix-neko@bolkedebruin@potiuk@uranusjr@hussein-awala@notatallshaw-gts