Support API for "pre-image" for pruning predicate evaluation by sdf-jkl · Pull Request #19722 · apache/datafusion

sdf-jkl · 2026-01-09T21:34:58Z

Which issue does this PR close?

closesSupport "pre-image" for pruning predicate evaluation #18320

Rationale for this change

Splitting the PR to make it more readable.

What changes are included in this PR?

Adding the udf_preimage logic without date_part implementation.

Are these changes tested?

Added unit tests for a test specific function

Are there any user-facing changes?

No

alamb

Thank you @sdf-jkl -- reviewed this PR carefully this morning and it looks great (thank you for splitting up the work), I found it well commented and well designed and a joy to read

I do think we need to add unit tests tests to for this feature, which I know you have lined up in #18789 but I think writing the unit tests in for the rewrite will make it easiest to validate.

I also have some questions about the rewrite for = (aka the boundary conditions)

alamb · 2026-01-20T17:21:36Z

I think this PR needs two more things:

Fi the NULL handling (probably by not calling preimage with null constants)
Update the API to have only a single method

(I am trying to keep my review context under control, so trying to focus on getting stuff through before starting more)

sdf-jkl · 2026-01-20T19:02:19Z

Both done. Re-requested a review.

alamb

Thank you very much @sdf-jkl

This looks good to me. I would like to change the signature to use Interval rather than Box<Interval> and there are a few other small comments, but we can also do this as a follow on PR (or I can push some commits to this PR)

Thank you for hanging with this one

FYI @colinmarc -- once we get this in, I think @sdf-jkl plans to implement preimage for date_part. Perhaps you are interested in something similar for date_trunc

Also, FYI @jonahgao and @xudong963 / @zhuqi-lucas in case you are interested in this PR (the primary usecase is improving the handling of date/timestamp predicates)

alamb · 2026-01-20T21:21:53Z

+ let Expr::ScalarFunction(ScalarFunction { func, args }) = left_expr else {
+ return Ok((None, None));
+ };
+ if !is_literal_or_literal_cast(right_expr) {


I think this is still an open question, but it is ok to handle as a follow on PR (aka widen the expressions)

alamb · 2026-01-20T21:24:24Z

+ if !is_literal_or_literal_cast(right_expr) {
+ return Ok(PreimageResult::None);
+ }
+ if func.signature().volatility != Volatility::Immutable {


Also for a follow on PR, I think it would be safe to rewrite stable functions (whose values don't change during the statement)

xudong963 · 2026-01-21T02:55:07Z

I'll have a look at the PR today

xudong963 · 2026-01-21T13:36:28Z

+/// [preimage]: https://en.wikipedia.org/wiki/Image_(mathematics)#Inverse_image
+///
+pub(super) fn rewrite_with_preimage(
+ _info: &SimplifyContext,


Do we need this arg?

@alamb mentioned that we should keep it in #18789 (comment), but it was a while ago.

I think it is important to pass to ScalarUDFImpl::preimage but probably it can be removed from this method call
Since I want to merge this PR up from main anyways before merge, I'll clean it up too

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

xudong963

Thank you for the work!

…tafusion into smaller-preimage-pr-1

alamb · 2026-01-22T21:49:44Z

Thank you @sdf-jkl and @xudong963 -- I took a final look through this and it looks really nice. Thank you.

I merged up from main and I'll plan to merge when the tests pass. Let's get this one in to keep things moving forward

alamb · 2026-01-22T23:01:58Z

FYI @rcurtin

alamb · 2026-01-22T23:02:06Z

Thanks again @sdf-jkl and @xudong963

rcurtin · 2026-01-22T23:10:52Z

Thank you for keeping me in the loop! 👍 Glad to see this.

…19722) ## Which issue does this PR close? - closesapache#18320 ## Rationale for this change Splitting the PR to make it more readable. ## What changes are included in this PR? Adding the udf_preimage logic without date_part implementation. ## Are these changes tested? Added unit tests for a test specific function ## Are there any user-facing changes? No --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Add udf_preimage logic
d94889a

github-actionsBot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Jan 9, 2026

sdf-jkland others added 7 commits January 9, 2026 20:37

Cargo fmt
4aa7f4e

Fix err in rewrite_with_preimage
2329c12

Rewrite the preimage_in_comparison
7ac8325

cargo fmt
7a3e8b3

Fix ci
fbd5dcc

Fix GtEq, Lt logic
d920735

Merge branch 'main' into smaller-preimage-pr-1
5ffb704

sdf-jkl mentioned this pull request Jan 10, 2026
Support "pre-image" for pruning predicate evaluation #18789
Closed
8 tasks

sdf-jkl mentioned this pull request Jan 18, 2026
Optimize the evaluation of DATE_TRUNC(<col>) == <constant>) when pushed down #18319
Open

alamb reviewed Jan 18, 2026
View reviewed changes

sdf-jkl force-pushed the smaller-preimage-pr-1 branch from f308662 to 5ffb704Compare January 18, 2026 18:21

github-actionsBot removed the documentation Improvements or additions to documentation label Jan 18, 2026

Fix comment
d8b4f0f

sdf-jkl added 4 commits January 20, 2026 12:50

Update API
116d6e2

clippy
c0ed63c

Fix null handling unit test
5856150

Fix null handling test
c53a9fc

sdf-jkl requested a review from alamb January 20, 2026 19:01

alamb approved these changes Jan 20, 2026
View reviewed changes

xudong963 reviewed Jan 21, 2026
View reviewed changes

sdf-jkland others added 4 commits January 21, 2026 09:27

Update datafusion/expr/src/preimage.rs
9b32843
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Fix docs
53f72ed

Fix comment
ba5be8a

Fix is_not_distinct_from rewrite
46a941f

xudong963 approved these changes Jan 22, 2026
View reviewed changes

alamb added 3 commits January 22, 2026 16:39

Merge remote-tracking branch 'apache/main' into smaller-preimage-pr-1
798d88f

Simplify the API
fb155f6

Merge branch 'smaller-preimage-pr-1' ofhttps://github.com/sdf-jkl/da…
a070246
…tafusion into smaller-preimage-pr-1

alamb mentioned this pull request Jan 22, 2026
Support "pre-image" for pruning predicate evaluation #18320
Closed

alamb added this pull request to the merge queue Jan 22, 2026

Merged via the queue into apache:main with commit c2f3d65Jan 22, 2026
32 checks passed

alamb mentioned this pull request Jan 27, 2026
implement preimage for date_trunc #18648
Closed

sdf-jkl mentioned this pull request Jan 28, 2026
Optimize the evaluation of date_part(<col>) == <constant> when pushed down #19733
Merged

Uh oh!

Conversation

sdf-jkl commented Jan 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Jan 20, 2026

Uh oh!

sdf-jkl commented Jan 20, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alambJan 20, 2026

Choose a reason for hiding this comment

Uh oh!

alambJan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xudong963 commented Jan 21, 2026

Uh oh!

xudong963Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

sdf-jklJan 21, 2026

Choose a reason for hiding this comment

Uh oh!

alambJan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xudong963 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

Uh oh!

rcurtin commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sdf-jkl commented Jan 9, 2026•
edited
Loading