Skip to content

Add support of Pendulum 3 - #36281

Merged
Taragolis merged 4 commits into
mainfrom
pendulum-3
Jan 12, 2024
Merged

Add support of Pendulum 3#36281
Taragolis merged 4 commits into
mainfrom
pendulum-3

Conversation

@Taragolis

@TaragolisTaragolis commented Dec 18, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Related: #35798, the differences is bump to min pendulum 3

As suggestion in https://lists.apache.org/thread/5b3xw87ntl0m00d8jwcz13vnvxhybjdp keep support pendulum 2 for a while, even if it has some problem it might help users to migrate to pendulum 3.

Add --downgrade-pendulum by the same way as it implements into the --downgrade-sqlalchemy so we could also run tests against pendulum 2 for a while.

The only one version of pendulum supported - 2.1.2 with older versions such as 2.0.0 I had a problem with initialised Airflow, there is should not be a big problem because we have 2.1.2 for a long period of time in constraints, and I guess for all Airflow 2.x but I do not checked all constraints only latests.

There is some different behaviour with pendulum 3 vs pendulum 2:

String representation:

Pendulum 2

Has T separator between date and time

>>>importpendulum>>>str(pendulum.now())
'2023-12-18T23:14:57.788852+04:00'

Pendulum 3

Has (1 whitespace) separator between date and time, same as datetime.datetime

>>>importpendulum>>>str(pendulum.now())
'2023-12-18 23:14:57.788852+04:00'

UTC serialised for timezone in REST API

Pendulum 2

"timezone": "Timezone('UTC')",

Pendulum 3

"timezone": "UTC",

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@bolkedebruin

bolkedebruin commented Dec 18, 2023

Copy link
Copy Markdown
Contributor

What happens if serialization (serde) encounters a OLD version of a pendulum timezone? I havent verified it myself but I assume the class names have changed?

@Taragolis

Taragolis commented Dec 18, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Full qualified path is the same. The main difference how UTC timezone works:

  • In pendulum 2 it pendulum.tz.timezone.FixedTimezone, so it deserialise into this type if it serialised in previous version nevermind we serialise offset=0 to UTC
  • In pendulum 3 it pendulum.tz.timezone.Timezone

@Taragolis

Taragolis commented Dec 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Anyway It is a good point to add this kind of tests.

@edgarrmondragonedgarrmondragon mentioned this pull request Dec 18, 2023
Closed
Taragolis
Taragolis commented Dec 18, 2023
View reviewed changes
Comment threadtests/api_connexion/endpoints/test_dag_endpoint.py Outdated
Comment threadtests/cli/commands/test_dag_command.py Outdated
uranusjr
uranusjr reviewed Dec 19, 2023
View reviewed changes
Comment threadairflow/example_dags/plugins/workday.py Outdated
@bolkedebruin

bolkedebruin commented Dec 21, 2023

Copy link
Copy Markdown
Contributor

I like it :-)

@TaragolisTaragolis mentioned this pull request Dec 23, 2023
Closed
@Taragolis

Taragolis commented Dec 23, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Seems like it close to backcompat with 2, just need to resolve remaining static checks

potiuk
potiuk reviewed Dec 24, 2023
View reviewed changes
Comment thread.github/workflows/ci.yml
@Taragolis

Taragolis commented Dec 24, 2023

Copy link
Copy Markdown
ContributorAuthor

Seems like all major stuff are resolved, so this PR could be reviewed.
The only things are required is add is newsfragment file

@Taragolis
Taragolis marked this pull request as ready for review December 24, 2023 11:55
@TaragolisTaragolis added this to the Airflow 2.8.1 milestone Dec 24, 2023
uranusjr
uranusjr reviewed Dec 24, 2023
View reviewed changes
Comment threadairflow/providers/cncf/kubernetes/pod_launcher_deprecated.py
uranusjr
uranusjr reviewed Dec 24, 2023
View reviewed changes
Comment threadairflow/serialization/serialized_objects.py Outdated
@potiuk

potiuk commented Dec 30, 2023

Copy link
Copy Markdown
Member

Looks green. Does it mean we have it ??

@potiukpotiuk mentioned this pull request Jan 2, 2024
Closed
2 tasks
@TaragolisTaragolis mentioned this pull request Jan 3, 2024
Closed
2 tasks
@TaragolisTaragolis mentioned this pull request Jan 4, 2024
Closed
2 tasks
@bolkedebruin

bolkedebruin commented Jan 6, 2024

Copy link
Copy Markdown
Contributor

I think we should bite the bullet and start fixing stuff that might be affected. So yes we should have it.

@Taragolis

Taragolis commented Jan 9, 2024

Copy link
Copy Markdown
ContributorAuthor

I will rebase and add release notes today if I feel better

@Taragolis

Taragolis commented Jan 10, 2024

Copy link
Copy Markdown
ContributorAuthor

Just wondering shall we merge it or better to wait for someone else review?

@Taragolis
Taragolisforce-pushed the pendulum-3 branch 2 times, most recently from ab31119 to 3caa58dCompareJanuary 10, 2024 21:49
@Taragolis

Taragolis commented Jan 11, 2024

Copy link
Copy Markdown
ContributorAuthor

Fortunately All failures after latest rebase not relevant to changes from this PR and should be fixed by #36728

potiuk
potiuk approved these changes Jan 11, 2024
View reviewed changes

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one looks FANTASTIC

potiuk
potiuk reviewed Jan 11, 2024
View reviewed changes
Comment threadscripts/docker/entrypoint_ci.sh
hussein-awala
hussein-awala approved these changes Jan 11, 2024
View reviewed changes

@hussein-awalahussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! LGTM

@potiukpotiuk mentioned this pull request Jan 11, 2024
Closed
2 tasks
@Taragolis
Taragolis merged commit 2ffa6e4 into mainJan 12, 2024
@Taragolis
Taragolis deleted the pendulum-3 branch January 12, 2024 10:27
@potiuk

potiuk commented Jan 12, 2024

Copy link
Copy Markdown
Member

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

potiuk pushed a commit that referenced this pull request Jan 13, 2024
@Taragolis@uranusjr@potiuk
Add support of Pendulum 3 (#36281)
0c93669
* Add support of Pendulum 3
* Add backcompat to pendulum 2
* Update airflow/serialization/serialized_objects.py
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
* Add newsfragments
---------
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit 2ffa6e4)
@ephraimbuddyephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Jan 15, 2024
ephraimbuddy pushed a commit that referenced this pull request Jan 15, 2024
@Taragolis@uranusjr@ephraimbuddy
Add support of Pendulum 3 (#36281)
dbc4b1f
* Add support of Pendulum 3
* Add backcompat to pendulum 2
* Update airflow/serialization/serialized_objects.py
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
* Add newsfragments
---------
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit 2ffa6e4)
@ephraimbuddyephraimbuddy mentioned this pull request Jan 16, 2024
Closed
68 tasks
@potiukpotiuk mentioned this pull request Jan 26, 2024
Closed
@sean-rosesean-rose mentioned this pull request Mar 1, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@uranusjruranusjruranusjr left review comments
@bolkedebruinbolkedebruinbolkedebruin approved these changes
@hussein-awalahussein-awalahussein-awala approved these changes
@ashbashbAwaiting requested review from ashbashb is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@kaxilkaxilAwaiting requested review from kaxil
@XD-DENGXD-DENGAwaiting requested review from XD-DENG

Assignees

No one assigned

Labels

area:serializationtype:misc/internalChangelog: Misc changes that should appear in change log

Projects

None yet

Milestone

Airflow 2.8.1

Development

Successfully merging this pull request may close these issues.

6 participants

@Taragolis@bolkedebruin@potiuk@uranusjr@hussein-awala@ephraimbuddy