Uh oh!
There was an error while loading. Please reload this page.
[SPARK-25784][SQL] Infer filters from constraints after rewriting predicate subquery - #22778
Closed
wangyum wants to merge 6 commits into
Closed
[SPARK-25784][SQL] Infer filters from constraints after rewriting predicate subquery#22778wangyum wants to merge 6 commits into
wangyum wants to merge 6 commits into
Conversation
SparkQA
commented
Oct 20, 2018
Test build #97629 has finished for PR 22778 at commit
|
SparkQA
commented
Oct 20, 2018
Test build #97655 has finished for PR 22778 at commit
|
wangyum
commented
Oct 20, 2018
MemberAuthor
retest this please |
SparkQA
commented
Oct 21, 2018
Test build #97669 has finished for PR 22778 at commit
|
wangyum
commented
Oct 21, 2018
MemberAuthor
maropu
commented
Oct 23, 2018
Member
Can you put the concrete example of the missing case you described in the PR description? |
maropu
commented
Oct 23, 2018
Member
Also, to make sure no performance regression in the optimizer, can you check optimizer statistics in TPCDS by running |
maropu
reviewed
Oct 23, 2018
| test("Column pruning after rewriting predicate subquery") { | ||
| val relation = LocalRelation('a.int, 'b.int) | ||
| val relInSubquery = LocalRelation('x.int, 'y.int, 'z.int) | ||
| withSQLConf(SQLConf.CONSTRAINT_PROPAGATION_ENABLED.key -> "false") { |
Member
There was a problem hiding this comment.
We need to modify this existing test?
MemberAuthor
There was a problem hiding this comment.
Yes, spark.sql.constraintPropagation.enabled=false to test ColumnPruning.spark.sql.constraintPropagation.enabled=true to test ColumnPruning, InferFiltersFromConstraints and PushDownPredicate.
maropu
reviewed
Oct 23, 2018
| } | ||
| } | ||
| test("Infer filters and push down predicate after rewriting predicate subquery") { |
Member
There was a problem hiding this comment.
Need the column pruning in the test title?
Member
There was a problem hiding this comment.
How about making the test title simple, then leaving comments about what's tested clearly here?
MemberAuthor
There was a problem hiding this comment.
How about refactor these test to:
valrelation=LocalRelation('a.int, 'b.int)
valrelInSubquery=LocalRelation('x.int, 'y.int, 'z.int)
test("Column pruning") {
withSQLConf(SQLConf.CONSTRAINT_PROPAGATION_ENABLED.key ->"false") {
valquery= relation.where('a.in(ListQuery(relInSubquery.select('x)))).select('a)
valoptimized=Optimize.execute(query.analyze)
valcorrectAnswer= relation
.select('a)
.join(relInSubquery.select('x), LeftSemi, Some('a==='x))
.analyze
comparePlans(optimized, correctAnswer)
}
}
test("Column pruning, infer filters and push down predicate") {
withSQLConf(SQLConf.CONSTRAINT_PROPAGATION_ENABLED.key ->"true") {
valquery= relation.where('a.in(ListQuery(relInSubquery.select('x)))).select('a)
valoptimized=Optimize.execute(query.analyze)
valcorrectAnswer= relation
.where(IsNotNull('a)).select('a)
.join(relInSubquery.where(IsNotNull('x)).select('x), LeftSemi, Some('a==='x))
.analyze
comparePlans(optimized, correctAnswer)
}
}| RewritePredicateSubquery, | ||
| ColumnPruning, | ||
| InferFiltersFromConstraints, | ||
| PushDownPredicate, |
wangyum
commented
Oct 25, 2018
MemberAuthor
@maropu This is optimizer statistics before and after this patch. |
wangyum
commented
Oct 29, 2018
MemberAuthor
We also need to add the |
| Batch("Check Cartesian Products", Once, | ||
| CheckCartesianProducts) :+ | ||
| Batch("RewriteSubquery", Once, | ||
| Batch("Rewrite Subquery", Once, |
Member
There was a problem hiding this comment.
I do not have a good answer for this PR. Ideally, we should run the whole batch operatorOptimizationBatch. However, running the whole batch could be very time consuming. I would suggest to add a new parameter for introducing the time bound limit for each batch.
cc @maryannxue WDYT?
Contributor
There was a problem hiding this comment.
@gatorsmile Do you think its a good time to revisit Natt's PR to convert subquery expressions to Joins early in the optimization process ? Perhaps then we can take advantage of all the subsequent rules firing after the subquery rewrite ?
Contributor
There was a problem hiding this comment.
@gatorsmile I think @dilipbiswal's suggestion is the right way to go. If you think of this subquery rewriting as another kind of de-correlation, it should be a pre-optimization rule.
Member
There was a problem hiding this comment.
Sure. That sounds also good to me. @dilipbiswal Could you take the PR #17520 over?
SparkQA
commented
Oct 30, 2018
Test build #98246 has finished for PR 22778 at commit
|
gatorsmile
commented
Oct 31, 2018
Member
Let us hold this PR and try to fix #17520 instead. |
SparkQA
commented
Oct 31, 2018
Test build #98294 has finished for PR 22778 at commit
|
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
SparkQA
commented
Aug 28, 2020
Test build #127986 has finished for PR 22778 at commit
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
SPARK-22662 fixed failed to prune columns after rewriting predicate subquery. but infer filters is still missing, this pr fix it.
Before this patch:
After this patch:
How was this patch tested?
unit tests and benchmark tests
1. Benchmark test 1
2. Benchmark test 2
This change affect TPC-DS: q4, q5, q8, q10, q11, q14a, q14b, q16, q23a, q23b, q33, q35, q56, q58, q60, q69, q70, q74, q83, q93, q94, q95, q5a, q10a, q14, q35, q35a, q70a, q74. and this is benchmark result(note that: you can click SQL name to compare the optimized plan before and after):