Change public simplify API and add a public coerce API by alamb · Pull Request #3758 · apache/datafusion

alamb · 2022-10-07T19:15:52Z

Which issue does this PR close?

Rationale for this change

I am not sure if anyone other than IOx calls the expression simplification directly, but we do and I have had to change it non trivially now that type coercion is required prior to simplification (b/c it is required to run the constant evaluator)

What changes are included in this PR?

Create a ExprSimplifier struct
Simplify and make public SimplfyContext
move the Expr simplification API to ExprSimplifier
Add a ExprSimplifier::coerce function and tests

Note I plan to add better documentation / examples via #3740 / #3741

Follow on API improvements proposed in #3793

Are there any user-facing changes?

New API, changes to existing simplify API (I will highlight the changes inline)

alamb · 2022-10-07T19:16:48Z

 // under the License.

-//! Expression simplifier
+//! Expression simplification API


This PR looks bigger than it really is -- it moves a bunch of code around and there are a significant number of docstrings

alamb · 2022-10-07T19:17:48Z

 fn execution_props(&self) -> &ExecutionProps;
 }

-/// trait for types that can be simplified


Removed in favor of calling ExprSimplifier::simplify directly

alamb · 2022-10-07T19:17:56Z

+ // rather than creating an DFSchemaRef coerces rather than doing
+ // it manually.
+ // TODO ticekt
+ pub fn coerce(&self, expr: Expr, schema: DFSchemaRef) -> Result<Expr> {


Here is the new API for coercion

ticekt, typo?

Good call -- will file ticket.

alamb · 2022-10-07T19:18:29Z

+/// let simplified = simplifier.simplify(expr).unwrap();
+/// assert_eq!(simplified, col("b").lt(lit(2)));
+/// ```
+pub struct SimplifyContext<'a> {


This was moved into the public API and I added some docs and a builder interface with_schema

alamb · 2022-10-07T19:18:39Z

 };
 use datafusion_physical_expr::{create_physical_expr, execution_props::ExecutionProps};

-/// Provides simplification information based on schema and properties


alamb · 2022-10-07T19:19:42Z

 };
 }

-/// Apply simplification and constant propagation to ([Expr]).


Instead of adding a simplify method on to Expr via this trait, I propose to have an ExprSimplifier struct that has a simplify method.
cc @ygf11
I found it made the examples in expr_api.rs less awkward to write because the schema wasn't needed

Looks great. It is more flexible, since user can define their SimplifyInfo other than inner SimplifyContext now.

alamb · 2022-10-07T19:25:12Z

+ /// See the [type coercion module](datafusion_expr::type_coercion)
+ /// documentation for more details on type coercion
+ ///
+ // Would be nice if this API could use the SimplifyInfo


If this PR is approved, I will file a ticket to remove the schema argument in this function call.

andygrove · 2022-10-11T00:46:11Z

I plan on reviewing this one tomorrow @alamb

alamb · 2022-10-11T10:06:40Z

I plan on reviewing this one tomorrow @alamb

Thank you @andygrove - I will have it ready and I'll also help try and review the outstanding DataFusion PRs as well

…coerce_api

alamb · 2022-10-11T16:16:00Z

This is blocking me upgrading DataFusion in my project (IOx). I know @andygrove plans to review today. Unless I hear otherwise I'll plan to merge PR tomorrow.

alamb · 2022-10-12T10:55:01Z

I am happy to respond to comments / make changes as a follow on PR

ursabot · 2022-10-12T11:03:34Z

Benchmark runs are scheduled for baseline = fb39d5d and contender = 61c38b7. 61c38b7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

alamb added the api change Changes the API exposed to users of the crate label Oct 7, 2022

github-actionsBot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Oct 7, 2022

alamb mentioned this pull request Oct 7, 2022
Introduce public API to call type coercion #3742
Closed

Public coerce and simplify API
56f115b

alamb force-pushed the alamb/new_simplify_coerce_api branch from 6567e5e to 56f115bCompare October 7, 2022 19:22

alamb commented Oct 7, 2022
View reviewed changes

github-actionsBot added the core Core DataFusion crate label Oct 7, 2022

alamb mentioned this pull request Oct 7, 2022
Add datafusion example of expression apis #3741
Merged
3 tasks

alamb changed the title ~~Change public simplify API and the Public coerce API~~Change public simplify API and add a public coerce APIOct 7, 2022

alamb commented Oct 7, 2022
View reviewed changes

andygrove self-requested a review October 11, 2022 00:53

Merge remote-tracking branch 'apache/master' into alamb/new_simplify_…
2365388
…coerce_api

alamb mentioned this pull request Oct 11, 2022
Improve coerce API so it doesn't need DF Schema #3793
Closed

alamb commented Oct 11, 2022
View reviewed changes

Comment threaddatafusion/optimizer/src/expr_simplifier.rs Outdated

alamb commented Oct 11, 2022
View reviewed changes

Comment threaddatafusion/optimizer/src/expr_simplifier.rs Outdated

Add ticket reference
5e1fd47

alamb merged commit 61c38b7 into apache:masterOct 12, 2022

alamb deleted the alamb/new_simplify_coerce_api branch October 12, 2022 10:54

Jefffrey mentioned this pull request Dec 29, 2025
Allow logical optimizer to be run without evaluating now() & refactor SimplifyInfo #19505
Merged

Uh oh!

Conversation

alamb commented Oct 7, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ygf11Oct 8, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ygf11Oct 8, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Oct 11, 2022

Uh oh!

alamb commented Oct 11, 2022

Uh oh!

Uh oh!

Uh oh!

alamb commented Oct 11, 2022

Uh oh!

alamb commented Oct 12, 2022

Uh oh!

ursabot commented Oct 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb commented Oct 7, 2022•
edited
Loading

ygf11Oct 8, 2022•
edited
Loading

ygf11Oct 8, 2022•
edited
Loading