Optimize the evaluation of date_part(<col>) == <constant> when pushed down by sdf-jkl · Pull Request #19733 · apache/datafusion

sdf-jkl · 2026-01-10T17:46:09Z

Which issue does this PR close?

closesOptimize the evaluation of date_part(<col>) == <constant> when pushed down #19889.

Rationale for this change

Check issue.

What changes are included in this PR?

Added preimage impl for date_part udf.

Added sqllogictests for the impl.

Are these changes tested?

Yes, sqllogictests.

Are there any user-facing changes?

No

…ler-preimage-pr-1

…tafusion into smaller-preimage-pr-2

…ler-preimage-pr-2

kosiew

LGTM

Left some minor suggestions.

kosiew · 2026-01-28T09:35:53Z

+ // date_part(col, YEAR) = 2024 => col >= '2024-01-01' and col < '2025-01-01'
+ // But for anything less than YEAR simplifying is not possible without specifying the bigger interval
+ // date_part(col, MONTH) = 1 => col = '2023-01-01' or col = '2024-01-01' or ... or col = '3000-01-01'
+ fn preimage(


Consider adding a section to docs/source/library-user-guide/functions/adding-udfs.md explaining:
What preimage is and when to implement it
How it enables predicate pushdown
Example implementation (perhaps referencing date_part)

There is a PR for preimage doc improvement here #20008.
I am however, not sure that this doc needs to explain preimage. I think the doc's goal is to be a very minimal guide on adding and registering a function. There is also no mention of simplify too.

I agree that the API docs is probably adequate. We could potentially add a note to adding-udfs.md that says something generic like "The ScalarUDFImpl has additional methods that support specialized optimizations such as preimage -- see the API documentation for additional details"

A separate PR should do.

sdf-jkl · 2026-01-28T16:45:04Z

Thanks for your review @kosiew. I've addressed your comments and also added proper handling for timestamps with timezones, please check it out too when you have time.

sdf-jkl · 2026-01-28T17:43:50Z

@alamb I'm still thinking about is_literal_or_literal_cast check in expr_simplifier: #19722 (comment)

alamb

Thank you @sdf-jkl -- this is looking very close

I have some suggestions on avoiding adding new date/time arithmetic here.

I also recommend an additional test for non-int32 arguments (coercion works fine, but it is good to show that):

diff --git a/datafusion/sqllogictest/test_files/udf_preimage.slt b/datafusion/sqllogictest/test_files/udf_preimage.slt
index 12f3a9a45b..c9d408227a 100644
--- a/datafusion/sqllogictest/test_files/udf_preimage.slt+++ b/datafusion/sqllogictest/test_files/udf_preimage.slt@@ -30,6 +30,11 @@ select c from t1 where extract(year from c) = 2024;
----
2024-01-01
+query D+select c from t1 where extract(year from c) = cast(2024 as bigint);+----+2024-01-01+
query D
select c from t1 where extract(year from c) <> 2024;
----
@@ -83,6 +88,16 @@ physical_plan
01)FilterExec: c@0 >= 2024-01-01 AND c@0 < 2025-01-01
02)--DataSourceExec: partitions=1, partition_sizes=[1]
+query TT+explain select c from t1 where extract (year from c) = cast(2024 as bigint)+----+logical_plan+01)Filter: t1.c >= Date32("2024-01-01") AND t1.c < Date32("2025-01-01")+02)--TableScan: t1 projection=[c]+physical_plan+01)FilterExec: c@0 >= 2024-01-01 AND c@0 < 2025-01-01+02)--DataSourceExec: partitions=1, partition_sizes=[1]+
query TT
explain select c from t1 where extract (year from c) <> 2024
----

alamb · 2026-01-29T18:13:17Z

+ // date_part(col, YEAR) = 2024 => col >= '2024-01-01' and col < '2025-01-01'
+ // But for anything less than YEAR simplifying is not possible without specifying the bigger interval
+ // date_part(col, MONTH) = 1 => col = '2023-01-01' or col = '2024-01-01' or ... or col = '3000-01-01'
+ fn preimage(


I agree that the API docs is probably adequate. We could potentially add a note to adding-udfs.md that says something generic like "The ScalarUDFImpl has additional methods that support specialized optimizations such as preimage -- see the API documentation for additional details"

alamb · 2026-01-29T18:28:42Z

+ Date32 => ScalarValue::Date32(Some(days as i32)),
+ Date64 => ScalarValue::Date64(Some(days * MILLISECONDS_IN_DAY)),
+
+ Timestamp(unit, tz_opt) => {


It feels to me like this code should be able to re-use the conversion functions in arrow rather than re-implementing them here
For example
Date32 => ScalarValue::Date32(Some(Date32Type::from_naive_date(date))),Date64 => ScalarValue::Date64(Some(Date64Type::from_naive_date(date))), ...
I didn't have a chance to figure out how to do it for Timestamp, but it seems like there should be a function like this for the timestamps too -- for example
https://docs.rs/arrow/latest/arrow/array/types/struct.TimestampSecondType.html and https://docs.rs/arrow/latest/arrow/datatypes/trait.ArrowTimestampType.html
Maybe something like
TimestampSecondType::make_value(date)
(We will have to figure out timestamps)

Fixed date32/64 62b0841
As for timestamp: dyn TimestampType::make_value() is using a NaiveDate, not DateTime<Tz>. We'd still have to do some tz math to create an offset for NaiveDate. (if only there was an existing API to help...)

Filed an upstream ticket:
Add some way to create a Timestamp from a DateTime arrow-rs#9337

alamb · 2026-02-02T23:20:16Z

+ lit_expr: &Expr,
+ info: &SimplifyContext,
+ ) -> Result<PreimageResult> {
+ let [part, col_expr] = take_function_args(self.name(), args)?;


I think we should be able to avoid writing all this code

alamb

Looks great to me -- thank you @sdf-jkl for pushing this over the line

alamb · 2026-02-03T16:55:58Z

Thanks again @kosiew for your help here

… down (apache#19733) ## Which issue does this PR close? - closesapache#19889. ## Rationale for this change Check issue. ## What changes are included in this PR? Added `preimage` impl for `date_part` udf. Added sqllogictests for the impl. ## Are these changes tested? Yes, sqllogictests. ## Are there any user-facing changes? No --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

sdf-jkland others added 10 commits January 9, 2026 16:32

Add udf_preimage logic
d94889a

Cargo fmt
4aa7f4e

Fix err in rewrite_with_preimage
2329c12

Rewrite the preimage_in_comparison
7ac8325

cargo fmt
7a3e8b3

Fix ci
fbd5dcc

Fix GtEq, Lt logic
d920735

Add datepart preimage + tests
d3318ff

Fix asf header
9fb245b

Merge branch 'main' into smaller-preimage-pr-1
5ffb704

github-actionsBot added logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jan 10, 2026

Merge branch 'main' into smaller-preimage-pr-2
372f704

sdf-jkl changed the title ~~Smaller preimage pr 2~~ Support "pre-image" for pruning predicate evaluation #2Jan 10, 2026

sdf-jkl mentioned this pull request Jan 10, 2026
Support "pre-image" for pruning predicate evaluation #18789
Closed
8 tasks

sdf-jkl mentioned this pull request Jan 18, 2026
Optimize the evaluation of DATE_TRUNC(<col>) == <constant>) when pushed down #18319
Open

sdf-jkl added 7 commits January 18, 2026 13:23

Merge branch 'main' ofhttps://github.com/apache/datafusioninto smal…
2fdc14c
…ler-preimage-pr-1

Replace BinaryExpression with binary_expr() fn
c2b0cd3

Add unit tests + add doc part about upper bound
a0b6564

Fix docs
0a24d60

Add datepart preimage + tests
86b7627

Fix asf header
0158662

Merge branch 'smaller-preimage-pr-2' ofhttps://github.com/sdf-jkl/da…
b491d4f
…tafusion into smaller-preimage-pr-2

github-actionsBot removed the core Core DataFusion crate label Jan 19, 2026

alamband others added 3 commits January 19, 2026 13:34

clippy
59235de

Merge remote-tracking branch 'apache/main' into smaller-preimage-pr-1
9ae434e

Make test field nullable
9f845e7

sdf-jkl changed the title ~~Support "pre-image" for pruning predicate evaluation #2~~Optimize the evaluation of date_part(<col>) == <constant> when pushed downJan 19, 2026

sdf-jkl added 6 commits January 19, 2026 15:32

Add datepart preimage + tests
08ef1f1

Fix asf header
57f6c4c

Merge branch 'smaller-preimage-pr-2' ofhttps://github.com/sdf-jkl/da…
e4dc727
…tafusion into smaller-preimage-pr-2

Merge branch 'main' ofhttps://github.com/apache/datafusioninto smal…
f36257e
…ler-preimage-pr-2

Fix date_part.rs
6bceb43

Fix udf_preimage.slt
13f1164

github-actionsBot removed logical-expr Logical plan and expressions optimizer Optimizer rules labels Jan 25, 2026

sdf-jkl added 2 commits January 25, 2026 15:54

Small fix
d902f65

fix typo
6992c8f

kosiew approved these changes Jan 28, 2026
View reviewed changes

sdf-jkl added 3 commits January 28, 2026 10:15

Add proper error handling
b456a22

Add tz slt tests
3669fb9

Add tz aware timestamp logic
bb6625f

sdf-jkl mentioned this pull request Jan 28, 2026
[Epic] Better support for DateTime XiangpengHao/liquid-cache#393
Open
4 tasks

alamb reviewed Jan 29, 2026
View reviewed changes

sdf-jkl added 2 commits January 29, 2026 15:17

Move tests to date_part.slt
aeafe1a

Use Arrow Type methods for date32/64
62b0841

sdf-jkl mentioned this pull request Feb 1, 2026
Implement preimage for floor function to enable predicate pushdown #20059
Merged

alamb reviewed Feb 2, 2026
View reviewed changes

alamb mentioned this pull request Feb 2, 2026
Add some way to create a Timestamp from a DateTimeapache/arrow-rs#9337
Closed

alamb approved these changes Feb 2, 2026
View reviewed changes

alamb added this pull request to the merge queue Feb 3, 2026

Merged via the queue into apache:main with commit 35e78caFeb 3, 2026
28 checks passed

Uh oh!

Conversation

sdf-jkl commented Jan 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sdf-jkl commented Jan 28, 2026

Uh oh!

sdf-jkl commented Jan 28, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jklJan 29, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Feb 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sdf-jkl commented Jan 10, 2026•
edited
Loading

sdf-jklJan 29, 2026•
edited
Loading