Skip to content

Support "pre-image" for pruning predicate evaluation - #18789

Closed
sdf-jkl wants to merge 40 commits into
apache:mainfrom
sdf-jkl:pre-image-support
Closed

Support "pre-image" for pruning predicate evaluation#18789
sdf-jkl wants to merge 40 commits into
apache:mainfrom
sdf-jkl:pre-image-support

Conversation

@sdf-jkl

@sdf-jklsdf-jkl commented Nov 17, 2025

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Adding a new rule to expr_simplifier library - udf_preimage

This rule performs the optimization for the following operators:

  • Equal
  • NotEqual
  • Greater
  • GreaterEqual
  • Less
  • LessEqual
  • IsDistinctFrom
  • IsNotDistinctFrom

The rule currently supports optimization for date_part function given 'year' literal is passed as the interval parameter.

Are these changes tested?

Tested for all comparison operators above and all possible datatypes in unit tests and sqllogictests.

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the optimizer Optimizer rules label Nov 17, 2025
@sdf-jkl

sdf-jkl commented Nov 17, 2025

Copy link
Copy Markdown
ContributorAuthor

@alamb This works as is right now, but I like the idea of adding a new method ScalarUDFImpl::simplify_predicate() as a unified api for all scalar functions. We then can move this code in the DatePartFunc implementation of ScalarUDFImpl and create a simplification rule for scalar functions.

The rule will check if there is a scalar function is present in the predicate expression and match the corresponding ScalarUDFImpl::simplify_predicate() to simplify the expression.

@2010YOUY01
2010YOUY01 self-requested a review November 18, 2025 04:49

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sdf-jkl -- this is looking quite cool

Comment threaddatafusion/optimizer/src/simplify_expressions/udf_preimage.rs Outdated

@2010YOUY012010YOUY01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is great 👍🏼

The implementation idea looks good to me. This should be good to go after the end-to-end tests are added, and also have the test coverage double checked.

Comment threaddatafusion/optimizer/src/simplify_expressions/expr_simplifier.rs Outdated
Comment threaddatafusion/optimizer/src/simplify_expressions/unwrap_date_part.rs Outdated
Comment threaddatafusion/optimizer/src/simplify_expressions/unwrap_date_part.rs Outdated
Comment threaddatafusion/optimizer/src/simplify_expressions/expr_simplifier.rs Outdated
@github-actionsgithub-actionsBot added logical-expr Logical plan and expressions functions Changes to functions implementation labels Nov 20, 2025
@github-actionsgithub-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Nov 21, 2025
@sdf-jkl

This comment was marked as outdated.

@github-actionsgithub-actionsBot added the core Core DataFusion crate label Dec 15, 2025
@sdf-jkl

Copy link
Copy Markdown
ContributorAuthor

Hey @alamb please check when you're available!

@sdf-jkl
sdf-jkl requested a review from alambDecember 15, 2025 18:05
@alamb

Copy link
Copy Markdown
Contributor

Thanks -- I'll try and find some time over the next day or two

@sdf-jkl

Copy link
Copy Markdown
ContributorAuthor

@alamb 👀

@sdf-jklsdf-jkl left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated IsDistinctFrom and IsNotDistinctFrom null handling logic and updated tests.

Comment threaddatafusion/core/tests/optimizer/mod.rs
@sdf-jkl

Copy link
Copy Markdown
ContributorAuthor

Hey @alamb, I've made some changes and ready for another go. Please take a look when you have time, thanks!

@alamb

alamb commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Sorry for the dealy here @sdf-jkl -- I am trying to find time to review this but I have been out and other things came up that were higher priority. I hope to return to this next week

@alamb

alamb commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

(it is hard for me to find time to review 1000 line PRs, unfortunately)

@sdf-jkl

Copy link
Copy Markdown
ContributorAuthor

@alamb no worries. Maybe I restructure this into some smaller PRs, so it will be easier to review?

@alamb

alamb commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

@alamb no worries. Maybe I restructure this into some smaller PRs, so it will be easier to review?

That would certainly help me 🙏

@sdf-jkl

sdf-jkl commented Jan 10, 2026

Copy link
Copy Markdown
ContributorAuthor

@alamb I split it in two:

Hope this will be helpful

@sdf-jkl

Copy link
Copy Markdown
ContributorAuthor

@sdf-jklsdf-jkl closed this Jan 19, 2026
@sdf-jkl
sdf-jkl deleted the pre-image-support branch January 19, 2026 20:31
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratefunctionsChanges to functions implementationlogical-exprLogical plan and expressionsoptimizerOptimizer rulessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support "pre-image" for pruning predicate evaluation

3 participants

@sdf-jkl@2010YOUY01@alamb