Make OptimizerConfig a trait (#4631) (#4638) by tustvold · Pull Request #4645 · apache/datafusion

tustvold · 2022-12-15T11:43:35Z

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

tustvold · 2022-12-15T11:44:34Z

 .register_catalog(config.default_catalog.clone(), default_catalog);
 }

- let optimizer_config = OptimizerConfig::new().filter_null_keys(


Moving this into the OptimizerConfig passed at optimize time is the fix for #4638

tustvold · 2022-12-15T11:45:30Z

+ Arc::new(CommonSubexprEliminate::new()),
+ Arc::new(PushDownProjection::new()),
 ];
- if config.filter_null_keys {


This is the other half of the fix for #4638 - determine the rules dynamically

tustvold · 2022-12-15T11:46:17Z

+ fn query_execution_start_time(&self) -> DateTime<Utc>;
+
+ /// Returns false if the given rule should be skipped
+ fn rule_enabled(&self, name: &str) -> bool;


I felt this formulation was more future proof than adding a specific function for enabling/disabling FilterNullJoins

tustvold · 2022-12-15T11:46:51Z

 Self {
 query_execution_start_time: Utc::now(),
- next_id: 0, // useful for generating things like unique subquery aliases
+ next_id: AtomicUsize::new(1),


We use 1, as previously the implementation returned the value post-increment which the atomics don't provide

tustvold · 2022-12-15T11:47:35Z

- // TODO this should not be on the config,
- // it should be its own 'OptimizerState' or something)
- next_id: usize,
+ next_id: AtomicUsize,


I think this limited interior mutability is better than the potential confusion around rules taking a mutable OptimizerConfig

I agree -- cc @waynexia

It is also good that OptimizerContext doesn't implement Clone which will discourage making copies that could get out of date

Looks good 👍

tustvold · 2022-12-15T11:48:44Z

- .with_query_execution_start_time(
- self.execution_props.query_execution_start_time,
- );
+ // TODO: Implement OptimizerContext directly on DataFrame (#4631) (#4626)


This is the ultimate motivation for this change, to allow a single config container, that then just implements the traits needed by the various sub-systems

I think having the sub crates depend on a traits that are implemented in the core trait sounds like a good idea to me

jackwener · 2022-12-15T13:38:59Z

I have some PR about optimizer😂, look like it will make some conflict.

I will wait for this PR merge, and then do them.

alamb

Looks like a (really) nice improvement to me -- thank you @tustvold 🏆

I think we should gather some more input and leave this open a while before merging it even though I think the actual required API change downstream are relatively low

cc @andygrove @jackwener @yahoNanJing @Dandandan @thinkharderdev @xudong963

alamb · 2022-12-15T18:38:30Z

- .with_query_execution_start_time(
- self.execution_props.query_execution_start_time,
- );
+ // TODO: Implement OptimizerContext directly on DataFrame (#4631) (#4626)


I think having the sub crates depend on a traits that are implemented in the core trait sounds like a good idea to me

alamb · 2022-12-15T18:41:31Z

 .register_catalog(config.default_catalog.clone(), default_catalog);
 }

- let optimizer_config = OptimizerConfig::new().filter_null_keys(


alamb · 2022-12-15T18:43:55Z

- // TODO this should not be on the config,
- // it should be its own 'OptimizerState' or something)
- next_id: usize,
+ next_id: AtomicUsize,


I agree -- cc @waynexia

alamb · 2022-12-15T18:44:59Z

- // TODO this should not be on the config,
- // it should be its own 'OptimizerState' or something)
- next_id: usize,
+ next_id: AtomicUsize,


It is also good that OptimizerContext doesn't implement Clone which will discourage making copies that could get out of date

xudong963

The change makes sense to me, thanks @tustvold

thinkharderdev

Makes sense to me

…trait

tustvold · 2022-12-16T15:24:39Z

I plan to merge this in the next few hours unless anybody objects / needs more time to review

ursabot · 2022-12-16T18:32:09Z

Benchmark runs are scheduled for baseline = 8944581 and contender = ca8985e. ca8985e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

alamb · 2022-12-16T19:35:22Z

🎉

really nice work @tustvold

tustvold added the api change Changes the API exposed to users of the crate label Dec 15, 2022

github-actionsBot added core Core DataFusion crate optimizer Optimizer rules labels Dec 15, 2022

tustvold commented Dec 15, 2022
View reviewed changes

tustvold requested a review from alamb December 15, 2022 11:48

Make OptimizerConfig a trait (apache#4631) (apache#4638)
babfb99

tustvold force-pushed the optimizer-config-trait branch from 8ce5714 to babfb99Compare December 15, 2022 12:48

Format
367c21e

alamb approved these changes Dec 15, 2022
View reviewed changes

xudong963 approved these changes Dec 16, 2022
View reviewed changes

thinkharderdev approved these changes Dec 16, 2022
View reviewed changes

tustvold added 2 commits December 16, 2022 15:22

Review feedback
bd9b3c9

Merge remote-tracking branch 'upstream/master' into optimizer-config-…
dbbcff5
…trait

tustvold merged commit ca8985e into apache:masterDec 16, 2022

Uh oh!

Conversation

tustvold commented Dec 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackwener commented Dec 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xudong963 left a comment

Choose a reason for hiding this comment

Uh oh!

thinkharderdev left a comment

Choose a reason for hiding this comment

Uh oh!

tustvold commented Dec 16, 2022

Uh oh!

ursabot commented Dec 16, 2022

Uh oh!

alamb commented Dec 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

tustvold commented Dec 15, 2022•
edited
Loading

jackwener commented Dec 15, 2022•
edited
Loading