[SPARK-42500][SQL] ConstantPropagation support more cases by peter-toth · Pull Request #40268 · apache/spark

peter-toth · 2023-03-03T14:39:34Z

What changes were proposed in this pull request?

This PR improves ConstantPropagation rule, the new implementation:

Collects constants into a map directly, instead of a list that is then transformed into a map.

Supports substituting in conflicting equalities:
e.g. a = 1 AND a = 2 => a = 1 AND 1 = 2
Before this PR this unsatisfiable filter was not reduced:

== Physical Plan ==
*(1) Project [value#219 AS a#222]
+- *(1) Filter ((isnotnull(value#219) AND (value#219 = 1)) AND (value#219 = 2))
+- *(1) LocalTableScan [value#219]

But after this PR it is:

== Physical Plan ==
LocalTableScan <empty>, [a#222]

Allows substitution in other than EqualTo and EqualNullSafeBinaryComparisonexpressions:
e.g. a = 5 AND b < a + 3 => a = 5 AND b < 8
Gets rid of the Option wrapper from the return type of ConstantPropagation.traverse() as modern .mapExpressions(), .withNewChildren() and .mapChildren() methods can recognize if there was no change in arguments.

Why are the changes needed?

Improve performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New UTs are taken from @wangyum's PR #40093

peter-toth · 2023-03-04T08:54:02Z

Tests look ok, removed WIP flag.

cc @wangyum, @cloud-fan

wangyum · 2023-03-06T10:59:38Z

+ // The keys are always canonicalized `AttributeReference`s, but it is easier to use `Expression`
+ // type keys here instead of casting `AttributeReference.canonicalized` to `AttributeReference` at
+ // the calling sites.
+ type EqualityPredicates = mutable.Map[Expression, (Literal, BinaryComparison)]


I think AttributeReference is enough.
Can we use AttributeMap ? In order to avoid the use of x.canonicalized later:
typeEqualityPredicates=AttributeMap[(Literal, BinaryComparison)]

Actually, I deliberately used mutable map here to improve their addition (equalityPredicates ++= equalityPredicatesRight) later.

I added a small improvement in cabb083

wangyum · 2023-03-06T11:05:01Z

+ private def replaceConstants(
+ condition: Expression,
+ equalityPredicates: EqualityPredicates): Expression = {
+ val predicates = equalityPredicates.values.map(_._2).toSet


Can we keep the constantsMap?
valconstantsMap=AttributeMap(equalityPredicates.map { case (attr, (lit, _)) => attr -> lit })

The reason for changing EqualityPredicates to mutable.Map earlier was to avoid building a map here.
The main point of that map is that we store only one Literal (and its original BinaryComparision) assigned to an attribute key. So if we have 2 or more conflicting EqualTo then in replaceConstants() we keep only one's original form and rewrite the other conflicing ones. E.g. a = 1 AND a = 2 we store only a -> (2, a = 2) in the map and rewrite the expression to 2 = 1 AND a = 2.

wangyum · 2023-03-06T11:05:48Z

+ case a: AttributeReference if equalityPredicates.contains(a.canonicalized) =>
+ equalityPredicates(a.canonicalized)._1


casea: AttributeReference=> constantsMap.getOrElse(a, a)
?

I kept the current mutable.Map based EqualityPredicates so I chage it to case a: AttributeReference => equalityPredicates.get(a.canonicalized).map(_._1).getOrElse(a).
But if we decide to use AttributeMap then I can change it to the suggested.

wangyum · 2023-03-07T00:15:17Z

 * resulted false
 * @return A tuple including:
- * 1. Option[Expression]: optional changed condition after traversal
+ * 1. Expression: optional changed condition after traversal


Remove the optional?

Thanks, I forgot to update that. Fixed in 0244034

peter-toth · 2023-03-10T17:13:51Z

@wangyum, @cloud-fan, do you think this PR is good enough or shall we go further with improvements?
The following features from #24553 are not in this PR currently to keep the change simple, but we can add them if needed:

Extend propagation from attribute => constant mapping to deterministic expression => constant mapping:
e.g. abs(a) = 5 AND b = abs(a) + 3 => abs(a) = 5 AND b = 8
Allow substitution in other than BinaryComparisons:
e.g. a = 5 AND abs(a) = 1 => a = 5 AND abs(5) = 1
Allow propagation of constant non-nullable expressions in Project and other nodes:
e.g. SELECT a = 5 AND b = a + 3 => SELECT a = 5 AND b = 8
(Currently only Filter is supported.)
Allows deep constant propagation:
e.g. WHERE IF(..., a = 5 AND b = a + 3, ...) => WHERE IF(..., a = 5 AND b = 8, ...)
(Currently only top level And/Or/Not are supported.)

@cloud-fan, I know you had concerns: #40093 (comment) and actually I'm not sure either how much performance improvement we could expect from the above 4 in real-life usecases...

…500-constantpropagation-support-more-cases

peter-toth · 2023-03-21T17:15:53Z

@cloud-fan, @wangyum please let me know if this PR needs further improvements.

wangyum · 2023-03-29T03:08:49Z

- } else {
- f
- }
+ f.mapExpressions(traverse(_, nullIsFalse = true, None))


This is our internal change:
objectConstantPropagationextendsRule[LogicalPlan] { defapply(plan: LogicalPlan):LogicalPlan= plan.transformUpWithPruning( _.containsAllPatterns(LITERAL, FILTER), ruleId) { casef: Filter=>val (newCondition, _) = traverse(f.condition, replaceChildren =true, nullIsFalse =true) if (newCondition.isDefined) { f.copy(condition = newCondition.get) } else { f } } /** * Traverse a condition as a tree and replace attributes with constant values. * - On matching [[And]], recursively traverse each children and get propagated mappings. * If the current node is not child of another [[And]], replace all occurrences of the * attributes with the corresponding constant values. * - If a child of [[And]] is [[EqualTo]] or [[EqualNullSafe]], propagate the mapping * of attribute => constant. * - On matching [[Or]] or [[Not]], recursively traverse each children, propagate empty mapping. * - Otherwise, stop traversal and propagate empty mapping. * @paramcondition condition to be traversed * @paramreplaceChildren whether to replace attributes with constant values in children * @paramnullIsFalse whether a boolean expression result can be considered to false e.g. in the * case of `WHERE e`, null result of expression `e` means the same as if it * resulted false * @return A tuple including: * 1. Option[Expression]: optional changed condition after traversal * 2. AttributeMap: propagated mapping of attribute => constant*/privatedeftraverse(condition: Expression, replaceChildren: Boolean, nullIsFalse: Boolean) : (Option[Expression], AttributeMap[(Literal, BinaryComparison)]) = condition match { case e @EqualTo(left: AttributeReference, right: Literal) if safeToReplace(left, nullIsFalse) => (None, AttributeMap(Map(left -> (right, e)))) case e @EqualTo(left: Literal, right: AttributeReference) if safeToReplace(right, nullIsFalse) => (None, AttributeMap(Map(right -> (left, e)))) case e @EqualNullSafe(left: AttributeReference, right: Literal) if safeToReplace(left, nullIsFalse) => (None, AttributeMap(Map(left -> (right, e)))) case e @EqualNullSafe(left: Literal, right: AttributeReference) if safeToReplace(right, nullIsFalse) => (None, AttributeMap(Map(right -> (left, e)))) casea: And=>val (newLeft, equalityPredicatesLeft) = traverse(a.left, replaceChildren =false, nullIsFalse) val (newRight, equalityPredicatesRight) = traverse(a.right, replaceChildren =false, nullIsFalse) valequalityPredicates= equalityPredicatesLeft ++ equalityPredicatesRight valnewSelf=if (equalityPredicates.nonEmpty && replaceChildren) { Some(And(replaceConstants(newLeft.getOrElse(a.left), equalityPredicates), replaceConstants(newRight.getOrElse(a.right), equalityPredicates))) } else { if (newLeft.isDefined || newRight.isDefined) { Some(And(newLeft.getOrElse(a.left), newRight.getOrElse(a.right))) } else { None } } (newSelf, equalityPredicates) caseo: Or=>// Ignore the EqualityPredicates from children since they are only propagated through And.val (newLeft, _) = traverse(o.left, replaceChildren =true, nullIsFalse) val (newRight, _) = traverse(o.right, replaceChildren =true, nullIsFalse) valnewSelf=if (newLeft.isDefined || newRight.isDefined) { Some(Or(left = newLeft.getOrElse(o.left), right = newRight.getOrElse((o.right)))) } else { None } (newSelf, AttributeMap.empty) casen: Not=>// Ignore the EqualityPredicates from children since they are only propagated through And.val (newChild, _) = traverse(n.child, replaceChildren =true, nullIsFalse =false) (newChild.map(Not), AttributeMap.empty) case _ => (None, AttributeMap.empty) } // We need to take into account if an attribute is nullable and the context of the conjunctive// expression. E.g. `SELECT * FROM t WHERE NOT(c = 1 AND c + 1 = 1)` where attribute `c` can be// substituted into `1 + 1 = 1` if 'c' isn't nullable. If 'c' is nullable then the enclosing// NOT prevents us to do the substitution as NOT flips the context (`nullIsFalse`) of what a// null result of the enclosed expression means.privatedefsafeToReplace(ar: AttributeReference, nullIsFalse: Boolean) =!ar.nullable || nullIsFalse privatedefreplaceConstants( condition: Expression, equalityPredicates: AttributeMap[(Literal, BinaryComparison)]):Expression= { valconstantsMap=AttributeMap(equalityPredicates.map { case (attr, (lit, _)) => attr -> lit }) valpredicates= equalityPredicates.values.map(_._2).toSet condition transform { caseb: BinaryComparisonif!predicates.contains(b) => b transform { casea: AttributeReference=> constantsMap.getOrElse(a, a) } } } }

I think this is very similar to a previous state of this PR but your version uses an immutable AttributeMap[(Literal, BinaryComparison)] instead of a mutable Map[Expression, (Literal, BinaryComparison)] in my PR. The addition of maps val equalityPredicates = equalityPredicatesLeft ++ equalityPredicatesRight inside handling Ands is slower when we use immutable maps. That's why I proposed to use a mutable one.
But since that version, I changed my PR to eliminate map addition at all and I use the mutable map as an accumulator from this commit: b9f3fbb. So the current version of this PR is even more optimized.

github-actions · 2023-07-08T00:22:09Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

peter-tothand others added 2 commits March 3, 2023 15:38

[SPARK-42500][SQL] ConstantPropagation support more cases
dc04923

SPARK-42500: ConstantPropagation support more case
0bb53d8

github-actionsBot added the SQL label Mar 3, 2023

peter-toth mentioned this pull request Mar 3, 2023
[SPARK-42500][SQL] ConstantPropagation supports more cases #40093
Closed

peter-toth changed the title ~~[WIP][SPARK-42500][SQL] ConstantPropagation support more cases~~[SPARK-42500][SQL] ConstantPropagation support more casesMar 4, 2023

wangyum reviewed Mar 6, 2023
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala Outdated

wangyum reviewed Mar 6, 2023
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala Outdated

wangyum reviewed Mar 6, 2023
View reviewed changes

fix review findings
1387247

wangyum reviewed Mar 7, 2023
View reviewed changes

peter-toth added 2 commits March 7, 2023 09:19

minor improvement
cabb083

fix scaladoc
0244034

improve constant map handling
b9f3fbb

peter-toth force-pushed the SPARK-42500-constantpropagation-support-more-cases branch from 99d6fea to b9f3fbbCompare March 13, 2023 08:34

peter-toth added 2 commits March 15, 2023 15:14

Merge commit '8860f69455e5a722626194c4797b4b42cccd4510' into SPARK-42…
9acb27e
…500-constantpropagation-support-more-cases

improve constant map handling 2
ecd650a

wangyum reviewed Mar 29, 2023
View reviewed changes

github-actionsBot added the Stale label Jul 8, 2023

github-actionsBot closed this Jul 9, 2023

peter-toth mentioned this pull request Jul 17, 2023
[SPARK-42500][SQL] ConstantPropagation support more case #42038
Closed

		case a: AttributeReference if equalityPredicates.contains(a.canonicalized) =>
		equalityPredicates(a.canonicalized)._1

Uh oh!

Conversation

peter-toth commented Mar 3, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

peter-toth commented Mar 4, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peter-tothMar 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peter-toth commented Mar 10, 2023

Uh oh!

peter-toth commented Mar 21, 2023

Uh oh!

wangyumMar 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peter-tothMar 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

peter-toth commented Mar 3, 2023•
edited
Loading

peter-tothMar 6, 2023•
edited
Loading

wangyumMar 29, 2023•
edited
Loading

peter-tothMar 29, 2023•
edited
Loading