Skip to content

Upgrade to DataFusion 53 - #1402

Merged
timsaucer merged 7 commits into
apache:mainfrom
nuno-faria:datafusion_53
Mar 6, 2026
Merged

Upgrade to DataFusion 53#1402
timsaucer merged 7 commits into
apache:mainfrom
nuno-faria:datafusion_53

Conversation

@nuno-faria

@nuno-farianuno-faria commented Mar 1, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A.

Rationale for this change

Keep in sync with the next major version of DataFusion.

What changes are included in this PR?

This PR is quite large, but most changes are related to the new version of PyO3.

  • Updated datafusion, pyo3, and arrow (migrated the respective code as well).
  • Updated some tests where the behavior changed.
  • Added small DataFusion 53.0.0 section in the upgrade guide (since PyCapsuleMethods::reference is now deprecated).

Are there any user-facing changes?

Yes, the behavior has changed in a few tests (the review comments below contain more info).

nuno-faria
nuno-faria commented Mar 1, 2026
View reviewed changes
Comment threaddocs/source/user-guide/upgrade-guides.rst Outdated
Comment threadexamples/datafusion-ffi-example/src/aggregate_udf.rs
Comment threadexamples/datafusion-ffi-example/src/catalog_provider.rs
Comment threadexamples/datafusion-ffi-example/src/utils.rs
Comment threadexamples/datafusion-ffi-example/src/utils.rs
Comment threadsrc/dataset_exec.rs
Comment threadsrc/dataset_exec.rs
Comment threadsrc/pyarrow_util.rs
Comment threadpython/tests/test_expr.py
Comment threadpython/tests/test_expr.py
@nuno-farianuno-faria mentioned this pull request Mar 1, 2026
Closed
26 tasks
@timsaucertimsaucer mentioned this pull request Mar 2, 2026
Closed
@timsaucer

timsaucer commented Mar 2, 2026

Copy link
Copy Markdown
Member

Nice work! There's quite a bit to review, but I plan on getting to it soon.

@timsaucer

timsaucer commented Mar 2, 2026

Copy link
Copy Markdown
Member

Related: #1407

@nuno-farianuno-faria mentioned this pull request Mar 4, 2026
Closed
@ntjohnson1ntjohnson1 mentioned this pull request Mar 5, 2026
Closed
timsaucer
timsaucer approved these changes Mar 6, 2026
View reviewed changes

@timsaucertimsaucer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for putting this together, and especially with the detailed sources for the diffs!

Comment threadpython/tests/test_functions.py
datetime(2020, 7, 1, tzinfo=DEFAULT_TZ),
],
type=pa.timestamp("ns", tz=DEFAULT_TZ),
type=pa.timestamp("us", tz=DEFAULT_TZ),

@timsaucertimsaucerMar 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think this is from apache/datafusion#19640 where it is now passing through the original data type from the input instead of changing it to nanoseconds. I think this is appropriate.

@timsaucer
timsaucer merged commit 8ef2cd7 into apache:mainMar 6, 2026
19 checks passed
@nuno-faria
nuno-faria deleted the datafusion_53 branch March 6, 2026 22:33
@kosiewkosiew mentioned this pull request Mar 17, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timsaucertimsaucertimsaucer approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nuno-faria@timsaucer