Skip to content

[SPARK-18589] [SQL] Fix Python UDF accessing attributes from both side of join - #16581

Closed
davies wants to merge 5 commits into
apache:masterfrom
davies:pyudf_join
Closed

[SPARK-18589] [SQL] Fix Python UDF accessing attributes from both side of join#16581
davies wants to merge 5 commits into
apache:masterfrom
davies:pyudf_join

Conversation

@davies

@daviesdavies commented Jan 13, 2017

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan.

This PR fix this issue by checking the expression is evaluable or not before pushing it into Join.

How was this patch tested?

Add a regression test.

@davies

davies commented Jan 13, 2017

Copy link
Copy Markdown
ContributorAuthor

cc @hvanhovell

@SparkQA

SparkQA commented Jan 13, 2017

Copy link
Copy Markdown

Test build #71346 has finished for PR 16581 at commit 95d73fc.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jan 14, 2017

Copy link
Copy Markdown

Test build #71349 has finished for PR 16581 at commit c45126b.

  • This patch fails from timeout after a configured wait of `250m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

rxin
rxin reviewed Jan 14, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala Outdated
case e: SubqueryExpression =>
// non-correlated subquery will be replaced as literal
e.children.nonEmpty
case e: Unevaluable => true

@rxinrxinJan 14, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need more documentation here on why should be considered evaluable as a join condition.

for example, just looking at this code i have no idea why Uneavaluable is evaluable.

@hvanhovellhvanhovellJan 14, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unevaluable is not evaluable. This block tries to find a case that is not evaluable in a join, and then negates it by isEmpty. I have to admit that we should document this.

rxin
rxin reviewed Jan 14, 2017
View reviewed changes
Comment threadpython/pyspark/sql/tests.py
df = df.withColumn('b', udf(lambda x: 'x')(df.a))
self.assertEqual(df.filter('b = "x"').collect(), [Row(a=1, b='x')])

def test_udf_in_filter_on_top_of_join(self):

@rxinrxinJan 14, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should reference jira number

davies
davies commented Jan 17, 2017
View reviewed changes
Comment threadpython/pyspark/sql/tests.py
df = df.withColumn('b', udf(lambda x: 'x')(df.a))
self.assertEqual(df.filter('b = "x"').collect(), [Row(a=1, b='x')])

def test_udf_in_filter_on_top_of_join(self):

@daviesdaviesJan 17, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@SparkQA

SparkQA commented Jan 17, 2017

Copy link
Copy Markdown

Test build #71520 has finished for PR 16581 at commit e4db820.

  • This patch fails from timeout after a configured wait of `250m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jan 19, 2017

Copy link
Copy Markdown

Test build #3541 has started for PR 16581 at commit e4db820.

@SparkQA

SparkQA commented Jan 19, 2017

Copy link
Copy Markdown

Test build #3542 has finished for PR 16581 at commit d6bba37.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jan 19, 2017

Copy link
Copy Markdown

Test build #71671 has finished for PR 16581 at commit d6bba37.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jan 20, 2017

Copy link
Copy Markdown

Test build #71681 has finished for PR 16581 at commit f720c85.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@hvanhovell

hvanhovell commented Jan 21, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

LGTM - merging to master.

@hvanhovell

hvanhovell commented Jan 21, 2017

Copy link
Copy Markdown
Contributor

I cannot backport it, could open a PR for 2.1.

asfgit pushed a commit that referenced this pull request Jan 21, 2017
Davies Liu@hvanhovell
[SPARK-18589][SQL] Fix Python UDF accessing attributes from both side…
4d286c9
… of join
PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan.
This PR fix this issue by checking the expression is evaluable or not before pushing it into Join.
Add a regression test.
Author: Davies Liu <davies@databricks.com>
Closes#16581 from davies/pyudf_join.
@davies

davies commented Jan 21, 2017

Copy link
Copy Markdown
ContributorAuthor

Cherry-picked into 2.1 branch.

uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
Davies Liu@uzadude
[SPARK-18589][SQL] Fix Python UDF accessing attributes from both side…
eb013be
… of join
## What changes were proposed in this pull request?
PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan.
This PR fix this issue by checking the expression is evaluable or not before pushing it into Join.
## How was this patch tested?
Add a regression test.
Author: Davies Liu <davies@databricks.com>
Closesapache#16581 from davies/pyudf_join.
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 15, 2017
Davies Liu@cmonkey
[SPARK-18589][SQL] Fix Python UDF accessing attributes from both side…
613986d
… of join
## What changes were proposed in this pull request?
PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan.
This PR fix this issue by checking the expression is evaluable or not before pushing it into Join.
## How was this patch tested?
Add a regression test.
Author: Davies Liu <davies@databricks.com>
Closesapache#16581 from davies/pyudf_join.
@xuanyuankingxuanyuanking mentioned this pull request Sep 4, 2018
Closed
asfgit pushed a commit that referenced this pull request Sep 27, 2018
@xuanyuanking@cloud-fan
[SPARK-25314][SQL] Fix Python UDF accessing attributes from both side…
53eb858
… of join in join conditions
## What changes were proposed in this pull request?
Thanks for bahchis reporting this. It is more like a follow up work for #16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition.
## How was this patch tested?
Add regression tests in PySpark and `BatchEvalPythonExecSuite`.
Closes#22326 from xuanyuanking/SPARK-25314.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 2a8cbfd)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
asfgit pushed a commit that referenced this pull request Sep 27, 2018
@xuanyuanking@cloud-fan
[SPARK-25314][SQL] Fix Python UDF accessing attributes from both side…
2a8cbfd
… of join in join conditions
## What changes were proposed in this pull request?
Thanks for bahchis reporting this. It is more like a follow up work for #16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition.
## How was this patch tested?
Add regression tests in PySpark and `BatchEvalPythonExecSuite`.
Closes#22326 from xuanyuanking/SPARK-25314.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
daspalrahul pushed a commit to daspalrahul/spark that referenced this pull request Sep 29, 2018
@xuanyuanking@daspalrahul
[SPARK-25314][SQL] Fix Python UDF accessing attributes from both side…
6785edd
… of join in join conditions
## What changes were proposed in this pull request?
Thanks for bahchis reporting this. It is more like a follow up work for apache#16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition.
## How was this patch tested?
Add regression tests in PySpark and `BatchEvalPythonExecSuite`.
Closesapache#22326 from xuanyuanking/SPARK-25314.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rxinrxinrxin left review comments
@hvanhovellhvanhovellhvanhovell left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@davies@SparkQA@hvanhovell@rxin