Skip to content

[SPARK-42500][SQL] ConstantPropagation support more case - #42038

Closed
TongWei1105 wants to merge 1 commit into
apache:masterfrom
TongWei1105:SPARK-42500
Closed

[SPARK-42500][SQL] ConstantPropagation support more case#42038
TongWei1105 wants to merge 1 commit into
apache:masterfrom
TongWei1105:SPARK-42500

Conversation

@TongWei1105

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR enhances ConstantPropagation to support more cases.

Propagated through other binary comparisons.
Propagated across equality comparisons. This can be further optimized to false.

Why are the changes needed?

Improve query performance. Denodo also has a similar optimization. For example:

CREATE TABLE t1(a int, b int) using parquet;
CREATE TABLE t2(x int, y int) using parquet;
CREATE TEMP VIEW v1 AS SELECT * FROM t1 JOIN t2 WHERE a = x AND a = 0 UNION ALL SELECT * FROM t1 JOIN t2 WHERE a = x AND (a IS NULL OR a <> 0);
SELECT * FROM v1 WHERE x > 1;

Before this PR:

== Optimized Logical Plan ==
Union false, false
:- Project [a#0 AS a#12, b#1 AS b#13, x#2 AS x#14, y#3 AS y#15]
: +- Join Inner
: :- Filter (isnotnull(a#0) AND (a#0 = 0))
: : +- Relation spark_catalog.default.t1[a#0,b#1] parquet
: +- Filter (isnotnull(x#2) AND ((0 = x#2) AND (x#2 > 1)))
: +- Relation spark_catalog.default.t2[x#2,y#3] parquet
+- Join Inner, (a#16 = x#18)
:- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
: +- Relation spark_catalog.default.t1[a#16,b#17] parquet
+- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
+- Relation spark_catalog.default.t2[x#18,y#19] parquet

After this PR:

== Optimized Logical Plan ==
Join Inner, (a#16 = x#18)
:- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
: +- Relation spark_catalog.default.t1[a#16,b#17] parquet
+- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
+- Relation spark_catalog.default.t2[x#18,y#19] parquet

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@TongWei1105
TongWei1105force-pushed the SPARK-42500 branch 3 times, most recently from 5712bb5 to ce86f25CompareJuly 17, 2023 08:59
@TongWei1105TongWei1105 changed the title SPARK-42500: ConstantPropagation support more case[SPARK-42500][SQL] ConstantPropagation support more caseJul 17, 2023
@wangyum

Copy link
Copy Markdown
Member

cc @cloud-fan@peter-toth

@peter-toth

Copy link
Copy Markdown
Contributor

I think this PR is very similar to my #40268 so I'm fine with this this change.
But my PR uses a mutable map to avoid the might be costly val equalityPredicates = equalityPredicatesLeft ++ equalityPredicatesRight map addition so please consider that PR too.

@TongWei1105
TongWei1105force-pushed the SPARK-42500 branch 3 times, most recently from 20d286f to 4d1e3a9CompareJuly 28, 2023 05:46
@wangyum

Copy link
Copy Markdown
Member

@peter-toth This map should usually be small. This change is small and easy to maintain.

@wangyum

Copy link
Copy Markdown
Member

Merged to master.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TongWei1105@wangyum@peter-toth