[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions by gatorsmile · Pull Request #10542 · apache/spark

gatorsmile · 2016-01-01T00:28:17Z

Conversion of outer joins, if the predicates in filter conditions can restrict the result sets so that all null-supplying rows are eliminated.

full outer -> inner if both sides have such predicates
left outer -> inner if the right side has such predicates
right outer -> inner if the left side has such predicates
full outer -> left outer if only the left side has such predicates
full outer -> right outer if only the right side has such predicates

If applicable, this can greatly improve the performance, since outer join is much slower than inner join, full outer join is much slower than left/right outer join.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in apache#10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closesapache#10543 from JoshRosen/dep-script-fixes.

SparkQA · 2016-01-01T04:52:44Z

Test build #48559 has finished for PR 10542 at commit 90576aa.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-01T06:36:17Z

Test build #48564 has finished for PR 10542 at commit 192ab19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

A following pr for apache#9712. Move the test for arrayOfUDT. Author: Liang-Chi Hsieh <viirya@gmail.com> Closesapache#10538 from viirya/move-udt-test.

A slight adjustment to the checker configuration was needed; there is a handful of warnings still left, but those are because of a bug in the checker that I'll fix separately (before enabling errors for the checker, of course). Author: Marcelo Vanzin <vanzin@cloudera.com> Closesapache#10535 from vanzin/SPARK-3873-mllib.

… for JDBCRDD and add few filters This patch refactors the filter pushdown for JDBCRDD and also adds few filters. Added filters are basically from apache#10468 with some refactoring. Test cases are from apache#10468. Author: Liang-Chi Hsieh <viirya@gmail.com> Closesapache#10470 from viirya/refactor-jdbc-filter.

rxin · 2016-01-01T09:03:10Z

is there a better name for this? this is a form of strength reduction right? I don't know if there is a better term in the database land. Can you look into postgres source code and see what they call this?

You are right. OuterJoinElimination might sound better. Let me rename it today.
Since full outer is union distinct of left outer and right outer, we are removing right outer from full outer when conversion from full outer to left outer.

There's a hack done in `TestHive.reset()`, which intended to mute noisy Hive loggers. However, Spark testing loggers are also muted. Author: Cheng Lian <lian@databricks.com> Closesapache#10540 from liancheng/spark-12592.dont-mute-spark-loggers.

…ut UnsafeRow It's confusing that some operator output UnsafeRow but some not, easy to make mistake. This PR change to only output UnsafeRow for all the operators (SparkPlan), removed the rule to insert Unsafe/Safe conversions. For those that can't output UnsafeRow directly, added UnsafeProjection into them. Closesapache#10330 cc JoshRosen rxin Author: Davies Liu <davies@databricks.com> Closesapache#10511 from davies/unsafe_row.

SparkQA · 2016-01-01T22:19:09Z

Test build #48569 has finished for PR 10542 at commit c04b53b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ays output UnsafeRow" This reverts commit 0da7bd5.

This PR inlines the Hive SQL parser in Spark SQL. The previous (merged) incarnation of this PR passed all tests, but had and still has problems with the build. These problems are caused by a the fact that - for some reason - in some cases the ANTLR generated code is not included in the compilation fase. This PR is a WIP and should not be merged until we have sorted out the build issues. Author: Herman van Hovell <hvanhovell@questtec.nl> Author: Nong Li <nong@databricks.com> Author: Nong Li <nongli@gmail.com> Closesapache#10525 from hvanhovell/SPARK-12362.

…ilter This PR is followed by apache#8391. Previous PR fixes JDBCRDD to support null-safe equality comparison for JDBC datasource. This PR fixes the problem that it can actually return null as a result of the comparison resulting error as using the value of that comparison. Author: hyukjinkwon <gurwls223@gmail.com> Author: HyukjinKwon <gurwls223@gmail.com> Closesapache#8743 from HyukjinKwon/SPARK-10180.

… APIs and reflection that supported 1.x Remove use of deprecated Hadoop APIs now that 2.2+ is required Author: Sean Owen <sowen@cloudera.com> Closesapache#10446 from srowen/SPARK-12481.

SparkQA · 2016-01-03T05:37:39Z

Test build #48597 has started for PR 10542 at commit 65f9125.

callUDF has been deprecated. However, we do not have an alternative for users to specify the output data type without type tags. This pull request introduced a new API for that, and replaces the invocation of the deprecated callUDF with that. Author: Reynold Xin <rxin@databricks.com> Closesapache#10547 from rxin/SPARK-12599.

…SQL] always output UnsafeRow"" This reverts commit 44ee920.

SparkQA · 2016-01-03T08:20:02Z

Test build #48600 has finished for PR 10542 at commit 0bb07cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-03T09:24:44Z

Test build #48604 has finished for PR 10542 at commit c5ff632.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

shivaram Author: felixcheung <felixcheung_m@hotmail.com> Closesapache#10408 from felixcheung/rcodecomment.

SparkQA · 2016-01-03T18:40:35Z

Test build #48609 has finished for PR 10542 at commit ee29dd2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Avoiding the the No such table exception and throwing analysis exception as per the bug: SPARK-12533 Author: thomastechs <thomas.sebastian@tcs.com> Closesapache#10529 from thomastechs/topic-branch.

gatorsmile · 2016-01-03T19:30:25Z

The failed bucket is not caused by the code changes in this PR. The code change is from the PR: #10275

SparkQA · 2016-01-03T21:48:18Z

Test build #48613 has finished for PR 10542 at commit ee29dd2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-03T22:06:48Z

Created a fix in #10564 , seems that the lag between the test being written and it getting merged had some bits change underneath it.

gatorsmile · 2016-01-03T22:22:09Z

Thank you! @holdenk

holdenk · 2016-01-03T22:24:59Z

Sure thing - thanks for pinging me when you noticed the issue :)

Author: Reynold Xin <rxin@databricks.com> Closesapache#10561 from rxin/update-mima.

…slash quoting mechanism We can provides the option to choose JSON parser can be enabled to accept quoting of all character or not. Author: Cazen <Cazen@korea.com> Author: Cazen Lee <cazen.lee@samsung.com> Author: Cazen Lee <Cazen@korea.com> Author: cazen.lee <cazen.lee@samsung.com> Closesapache#10497 from Cazen/master.

Previously (when the PR was first created) not specifying b= explicitly was fine (and treated as default null) - instead be explicit about b being None in the test. Author: Holden Karau <holden@us.ibm.com> Closesapache#10564 from holdenk/SPARK-12611-fix-test-infer-schema-local.

…oinElimination # Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

gatorsmile · 2016-01-04T03:17:32Z

Let me close it and then resubmit the PR.

SparkQA · 2016-01-04T06:52:51Z

Test build #48628 has finished for PR 10542 at commit ee29dd2.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-04T07:33:39Z

Test build #48632 has finished for PR 10542 at commit 63d5d62.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

Conversion of outer joins, if the predicates in filter conditions can restrict the result sets so that all null-supplying rows are eliminated. - `full outer` -> `inner` if both sides have such predicates - `left outer` -> `inner` if the right side has such predicates - `right outer` -> `inner` if the left side has such predicates - `full outer` -> `left outer` if only the left side has such predicates - `full outer` -> `right outer` if only the right side has such predicates If applicable, this can greatly improve the performance, since outer join is much slower than inner join, full outer join is much slower than left/right outer join. The original PR is #10542 Author: gatorsmile <gatorsmile@gmail.com> Author: xiaoli <lixiao1983@gmail.com> Author: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local> Closes#10567 from gatorsmile/outerJoinEliminationByFilterCond.

gatorsmileand others added 4 commits December 31, 2015 15:48

Merge remote-tracking branch 'upstream/master' into outerJoinElimination
2f6c80d

outer join conversion
90576aa

added test cases.
192ab19

viiryaand others added 3 commits December 31, 2015 23:48

[SPARK-11743][SQL] Move the test for arrayOfUDT
c9dbfcc
A following pr for apache#9712. Move the test for arrayOfUDT. Author: Liang-Chi Hsieh <viirya@gmail.com> Closesapache#10538 from viirya/move-udt-test.

rxin reviewed Jan 1, 2016
View reviewed changes

renaming
c04b53b

gatorsmile changed the title ~~[SPARK-12594] [SQL] Outer Join Conversion: Outer/Right/Left to Inner and Outer to Left/Right~~[SPARK-12594] [SQL] Outer Join Elimination: Outer/Right/Left to Inner and Outer to Left/RightJan 1, 2016

lianchengand others added 3 commits January 1, 2016 13:24

Disable test-dependencies.sh.
6c20b3c

Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][SQL] alw…
44ee920
…ays output UnsafeRow" This reverts commit 0da7bd5.

gatorsmile changed the title ~~[SPARK-12594] [SQL] Outer Join Elimination: Outer/Right/Left to Inner and Outer to Left/Right~~[SPARK-12594] [SQL] Outer Join Elimination by Local PredicatesJan 2, 2016

hvanhovelland others added 4 commits January 1, 2016 23:22

[SPARK-12481][CORE][STREAMING][SQL] Remove usage of Hadoop deprecated…
15bd736
… APIs and reflection that supported 1.x Remove use of deprecated Hadoop APIs now that 2.2+ is required Author: Sean Owen <sowen@cloudera.com> Closesapache#10446 from srowen/SPARK-12481.

extend the condition to cover more cases in non null predicates.
65f9125

rxinand others added 4 commits January 2, 2016 22:31

Revert "Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][…
6c5bbd6
…SQL] always output UnsafeRow"" This reverts commit 44ee920.

added three more expressions: and, or and not
9398644

style fix.
0bb07cb

support non-local predicates and bug fix.
c5ff632

gatorsmile changed the title ~~[SPARK-12594] [SQL] Outer Join Elimination by Local Predicates~~[SPARK-12594] [SQL] Outer Join Elimination by Filter ConditionsJan 3, 2016

felixcheungand others added 2 commits January 3, 2016 20:53

[SPARK-12327][SPARKR] fix code for lintr warning for commented code
c3d5056
shivaram Author: felixcheung <felixcheung_m@hotmail.com> Closesapache#10408 from felixcheung/rcodecomment.

scala style fix.
ee29dd2

[SPARK-12533][SQL] hiveContext.table() throws the wrong exception
c82924d
Avoiding the the No such table exception and throwing analysis exception as per the bug: SPARK-12533 Author: thomastechs <thomas.sebastian@tcs.com> Closesapache#10529 from thomastechs/topic-branch.

rxinand others added 3 commits January 3, 2016 16:58

Update MimaExcludes now Spark 1.6 is in Maven.
7b92922
Author: Reynold Xin <rxin@databricks.com> Closesapache#10561 from rxin/update-mima.

gatorsmile mentioned this pull request Jan 4, 2016
[SPARK-12613] [SQL] Outer Join Elimination by Parent Join Condition #10566
Closed

gatorsmile added 10 commits January 3, 2016 18:54

outer join conversion
7b7ea90

added test cases.
7558e70

renaming
d3cbf46

extend the condition to cover more cases in non null predicates.
2535cb1

added three more expressions: and, or and not
6c3f4b0

style fix.
fcd757c

support non-local predicates and bug fix.
5bc7f52

scala style fix.
34a0056

code refactoring and code merge
ee7db1a

Merge remote-tracking branch 'origin/outerJoinConversion' into outerJ…
63d5d62
…oinElimination # Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

gatorsmile closed this Jan 4, 2016

gatorsmile mentioned this pull request Jan 4, 2016
[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions #10567
Closed

Uh oh!

Conversation

gatorsmile commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!

rxinJan 1, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmileJan 1, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 3, 2016

Uh oh!

SparkQA commented Jan 3, 2016

Uh oh!

SparkQA commented Jan 3, 2016

Uh oh!

SparkQA commented Jan 3, 2016

Uh oh!

gatorsmile commented Jan 3, 2016

Uh oh!

SparkQA commented Jan 3, 2016

Uh oh!

holdenk commented Jan 3, 2016

Uh oh!

gatorsmile commented Jan 3, 2016

Uh oh!

holdenk commented Jan 3, 2016

Uh oh!

gatorsmile commented Jan 4, 2016

Uh oh!

SparkQA commented Jan 4, 2016

Uh oh!

SparkQA commented Jan 4, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants