[refactor]: Convert Vec<PhysicalExpr> to HashSet<PhysicalExpr> by akurmustafa · Pull Request #13612 · apache/datafusion

akurmustafa · 2024-12-01T05:49:06Z

Which issue does this PR close?

Closes #.

Rationale for this change

Vec<Arc<dynPhysicalExpr>>s inside EquivalenceClass is conceptually set. Previously, since Arc<dyn PhysicalExpr> didnot support Eq, we couldn't use it insideHashSet (or IndexSet). With recent changes in Datafusion, we can do so now. This PR converts Vec<Arc<dynPhysicalExpr>> into IndexSet<Arc<dynPhysicalExpr>> to make implementation simpler.

What changes are included in this PR?

Are these changes tested?

Existing tests

Are there any user-facing changes?

Weijun-H

Nice improvement! Thanks @akurmustafa 👍

Weijun-H · 2024-12-01T12:06:00Z

@@ -191,46 +190,48 @@ pub struct EquivalenceClass {
 /// matter for equivalence purposes
 ///
 /// TODO: use a HashSet for this instead of a Vec


please remove this comment

Thanks, missed it

Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

alamb

This looks awesome -- thanks @akurmustafa and @Weijun-H -- I am also running some benchmarks to see if it improves performance for planning

alamb · 2024-12-02T18:46:09Z

- Self { exprs }
+ pub fn new(exprs: Vec<Arc<dyn PhysicalExpr>>) -> Self {
+ Self {
+ exprs: exprs.into_iter().collect(),


Weijun-H · 2024-12-03T11:14:33Z

Thanks @akurmustafa and @alamb

alamb · 2024-12-03T13:04:43Z

Hmm, I seem to see a slight slow down in planning 🤔 maybe worth some profiling

++ critcmp main refactor_hashset_exprs
group main refactor_hashset_exprs
----- ---- ----------------------
logical_aggregate_with_join 1.00 1471.3±32.58µs ? ?/sec 1.01 1484.4±20.25µs ? ?/sec
logical_select_all_from_1000 1.00 5.2±0.02ms ? ?/sec 1.00 5.2±0.03ms ? ?/sec
logical_select_one_from_700 1.00 1177.1±16.70µs ? ?/sec 1.00 1172.2±16.21µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 1137.3±16.03µs ? ?/sec 1.01 1144.0±13.67µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 1122.4±16.65µs ? ?/sec 1.02 1144.8±25.55µs ? ?/sec
physical_intersection 1.00 2.4±0.04ms ? ?/sec 1.00 2.4±0.04ms ? ?/sec
physical_join_consider_sort 1.00 3.2±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_join_distinct 1.01 1122.7±16.06µs ? ?/sec 1.00 1117.1±17.36µs ? ?/sec
physical_many_self_joins 1.00 17.0±0.09ms ? ?/sec 1.00 17.0±0.11ms ? ?/sec
physical_plan_clickbench_all 1.00 225.4±1.82ms ? ?/sec 1.01 227.7±1.99ms ? ?/sec
physical_plan_clickbench_q1 1.00 3.3±0.04ms ? ?/sec 1.01 3.3±0.05ms ? ?/sec
physical_plan_clickbench_q10 1.00 4.3±0.05ms ? ?/sec 1.01 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q11 1.00 4.5±0.06ms ? ?/sec 1.01 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q12 1.00 4.6±0.08ms ? ?/sec 1.01 4.6±0.06ms ? ?/sec
physical_plan_clickbench_q13 1.00 4.2±0.05ms ? ?/sec 1.01 4.2±0.06ms ? ?/sec
physical_plan_clickbench_q14 1.00 4.4±0.07ms ? ?/sec 1.01 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q15 1.00 4.3±0.06ms ? ?/sec 1.03 4.4±0.17ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.7±0.04ms ? ?/sec 1.01 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.8±0.06ms ? ?/sec 1.01 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q18 1.00 3.6±0.04ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.00 4.5±0.05ms ? ?/sec 1.01 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q2 1.00 3.6±0.05ms ? ?/sec 1.01 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q20 1.00 3.3±0.05ms ? ?/sec 1.01 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q21 1.00 3.6±0.05ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q22 1.00 4.6±0.08ms ? ?/sec 1.01 4.6±0.07ms ? ?/sec
physical_plan_clickbench_q23 1.00 5.0±0.06ms ? ?/sec 1.01 5.0±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.00 5.8±0.07ms ? ?/sec 1.01 5.8±0.09ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.9±0.05ms ? ?/sec 1.01 4.0±0.08ms ? ?/sec
physical_plan_clickbench_q26 1.00 3.6±0.06ms ? ?/sec 1.02 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q27 1.00 4.0±0.06ms ? ?/sec 1.01 4.0±0.05ms ? ?/sec
physical_plan_clickbench_q28 1.00 4.7±0.07ms ? ?/sec 1.02 4.8±0.11ms ? ?/sec
physical_plan_clickbench_q29 1.00 5.8±0.07ms ? ?/sec 1.01 5.9±0.10ms ? ?/sec
physical_plan_clickbench_q3 1.00 3.5±0.05ms ? ?/sec 1.02 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q30 1.00 16.6±0.21ms ? ?/sec 1.00 16.6±0.18ms ? ?/sec
...

Weijun-H · 2024-12-04T01:52:40Z

Hmm, I seem to see a slight slow down in planning 🤔 maybe worth some profiling

++ critcmp main refactor_hashset_exprs
group main refactor_hashset_exprs
----- ---- ----------------------
logical_aggregate_with_join 1.00 1471.3±32.58µs ? ?/sec 1.01 1484.4±20.25µs ? ?/sec
logical_select_all_from_1000 1.00 5.2±0.02ms ? ?/sec 1.00 5.2±0.03ms ? ?/sec
logical_select_one_from_700 1.00 1177.1±16.70µs ? ?/sec 1.00 1172.2±16.21µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 1137.3±16.03µs ? ?/sec 1.01 1144.0±13.67µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 1122.4±16.65µs ? ?/sec 1.02 1144.8±25.55µs ? ?/sec
physical_intersection 1.00 2.4±0.04ms ? ?/sec 1.00 2.4±0.04ms ? ?/sec
physical_join_consider_sort 1.00 3.2±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_join_distinct 1.01 1122.7±16.06µs ? ?/sec 1.00 1117.1±17.36µs ? ?/sec
physical_many_self_joins 1.00 17.0±0.09ms ? ?/sec 1.00 17.0±0.11ms ? ?/sec
physical_plan_clickbench_all 1.00 225.4±1.82ms ? ?/sec 1.01 227.7±1.99ms ? ?/sec
physical_plan_clickbench_q1 1.00 3.3±0.04ms ? ?/sec 1.01 3.3±0.05ms ? ?/sec
physical_plan_clickbench_q10 1.00 4.3±0.05ms ? ?/sec 1.01 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q11 1.00 4.5±0.06ms ? ?/sec 1.01 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q12 1.00 4.6±0.08ms ? ?/sec 1.01 4.6±0.06ms ? ?/sec
physical_plan_clickbench_q13 1.00 4.2±0.05ms ? ?/sec 1.01 4.2±0.06ms ? ?/sec
physical_plan_clickbench_q14 1.00 4.4±0.07ms ? ?/sec 1.01 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q15 1.00 4.3±0.06ms ? ?/sec 1.03 4.4±0.17ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.7±0.04ms ? ?/sec 1.01 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.8±0.06ms ? ?/sec 1.01 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q18 1.00 3.6±0.04ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.00 4.5±0.05ms ? ?/sec 1.01 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q2 1.00 3.6±0.05ms ? ?/sec 1.01 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q20 1.00 3.3±0.05ms ? ?/sec 1.01 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q21 1.00 3.6±0.05ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q22 1.00 4.6±0.08ms ? ?/sec 1.01 4.6±0.07ms ? ?/sec
physical_plan_clickbench_q23 1.00 5.0±0.06ms ? ?/sec 1.01 5.0±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.00 5.8±0.07ms ? ?/sec 1.01 5.8±0.09ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.9±0.05ms ? ?/sec 1.01 4.0±0.08ms ? ?/sec
physical_plan_clickbench_q26 1.00 3.6±0.06ms ? ?/sec 1.02 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q27 1.00 4.0±0.06ms ? ?/sec 1.01 4.0±0.05ms ? ?/sec
physical_plan_clickbench_q28 1.00 4.7±0.07ms ? ?/sec 1.02 4.8±0.11ms ? ?/sec
physical_plan_clickbench_q29 1.00 5.8±0.07ms ? ?/sec 1.01 5.9±0.10ms ? ?/sec
physical_plan_clickbench_q3 1.00 3.5±0.05ms ? ?/sec 1.02 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q30 1.00 16.6±0.21ms ? ?/sec 1.00 16.6±0.18ms ? ?/sec
...

track #13638

akurmustafa added 2 commits November 30, 2024 10:16

Initial commit
47d55c6

Change implementation to take iterator
7857872

github-actionsBot added physical-expr Changes to the physical-expr crates common Related to common crate labels Dec 1, 2024

Minor changes
3cdc42a

github-actionsBot removed the common Related to common crate label Dec 1, 2024

Weijun-H approved these changes Dec 1, 2024
View reviewed changes

akurmustafaand others added 2 commits December 1, 2024 12:33

Update datafusion/physical-expr/src/equivalence/class.rs
fa65753
Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

Remove leftover comment
643dcef

alamb approved these changes Dec 2, 2024
View reviewed changes

Weijun-H merged commit fdb221f into apache:mainDec 3, 2024

Weijun-H mentioned this pull request Dec 4, 2024
Planning performance regression after replacing Vec<PhysicalExpr> with HashSet<PhysicalExpr>. #13638
Open

Navigation Menu

Uh oh!

[refactor]: Convert Vec<PhysicalExpr> to HashSet<PhysicalExpr> - #13612

[refactor]: Convert Vec<PhysicalExpr> to HashSet<PhysicalExpr>#13612
Weijun-H merged 5 commits into
apache:mainfrom
akurmustafa:refactor/hashset_exprs

akurmustafa commented Dec 1, 2024

Uh oh!

Weijun-H left a comment

Uh oh!

Weijun-HDec 1, 2024

Uh oh!

akurmustafaDec 1, 2024

Uh oh!

Uh oh!

alamb left a comment

Uh oh!

alambDec 2, 2024

Uh oh!

Weijun-H commented Dec 3, 2024

Uh oh!

alamb commented Dec 3, 2024

Uh oh!

Weijun-H commented Dec 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

akurmustafa commented Dec 1, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Weijun-H left a comment

Choose a reason for hiding this comment

Uh oh!

Weijun-HDec 1, 2024

Choose a reason for hiding this comment

Uh oh!

akurmustafaDec 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambDec 2, 2024

Choose a reason for hiding this comment

Uh oh!

Weijun-H commented Dec 3, 2024

Uh oh!

alamb commented Dec 3, 2024

Uh oh!

Weijun-H commented Dec 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants