Skip to content

[SPARK-28321][SQL] 0-args Java UDF should not be called only once - #25108

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-28321
Closed

[SPARK-28321][SQL] 0-args Java UDF should not be called only once#25108
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-28321

Conversation

@HyukjinKwon

@HyukjinKwonHyukjinKwon commented Jul 11, 2019•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

0-args Java UDF alone calls the function even before making it as an expression.
It causes that the function always returns the same value and the function is called at driver side.
Seems like a mistake.

How was this patch tested?

Unit test was added

@HyukjinKwon

HyukjinKwon commented Jul 11, 2019

Copy link
Copy Markdown
MemberAuthor

cc @cloud-fan

@cloud-fan

cloud-fan commented Jul 11, 2019

Copy link
Copy Markdown
Contributor

good catch! LGTM

@SparkQA

SparkQA commented Jul 11, 2019

Copy link
Copy Markdown

Test build #107507 has finished for PR 25108 at commit c725002.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Jul 11, 2019

Copy link
Copy Markdown
MemberAuthor

retest this please

viirya
viirya approved these changes Jul 11, 2019
View reviewed changes

@viiryaviirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

viirya
viirya reviewed Jul 11, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/functions.scala Outdated
HyukjinKwon
HyukjinKwon commented Jul 11, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala Outdated

@HyukjinKwonHyukjinKwonJul 11, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one, I manually tested but didn't add the test just in case we want to add some kind of optimization in the future. We shouldn't do such thing here but in optimizer anyway. Seems like just a mistake.

@SparkQA

SparkQA commented Jul 11, 2019

Copy link
Copy Markdown

Test build #107517 has finished for PR 25108 at commit c725002.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

viirya
viirya reviewed Jul 11, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala Outdated
@SparkQA

SparkQA commented Jul 11, 2019

Copy link
Copy Markdown

Test build #107529 has finished for PR 25108 at commit 9039f3d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Jul 11, 2019
View reviewed changes

@srowensrowen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK, though I don't know enough to really review.

@SparkQA

SparkQA commented Jul 11, 2019

Copy link
Copy Markdown

Test build #107536 has finished for PR 25108 at commit 1567b6c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Jul 12, 2019

Copy link
Copy Markdown
Contributor

thanks, merging to master!

vinodkc pushed a commit to vinodkc/spark that referenced this pull request Jul 18, 2019
@HyukjinKwon@vinodkc
[SPARK-28321][SQL] 0-args Java UDF should not be called only once
6280475
## What changes were proposed in this pull request?
0-args Java UDF alone calls the function even before making it as an expression.
It causes that the function always returns the same value and the function is called at driver side.
Seems like a mistake.
## How was this patch tested?
Unit test was added
Closesapache#25108 from HyukjinKwon/SPARK-28321.
Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@gatorsmile

gatorsmile commented Jul 21, 2019

Copy link
Copy Markdown
Member

This a nice fix, but we still need to document it in the migration guide. This changes a behavior. Also it could cause a perf regression when calling the 0-args Java UDF is expensive.

@HyukjinKwon

HyukjinKwon commented Jul 22, 2019

Copy link
Copy Markdown
MemberAuthor

Yea, that's fine. Let me update the migration guide.

@HyukjinKwonHyukjinKwon mentioned this pull request Jul 22, 2019
Closed
yiheng pushed a commit to yiheng/spark that referenced this pull request Jul 24, 2019
@HyukjinKwon@yiheng
[SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java …
3ed547e
…UDF's internal behaviour change
## What changes were proposed in this pull request?
This PR proposes to add a note in the migration guide. See apache#25108 (comment)
## How was this patch tested?
N/A
Closesapache#25224 from HyukjinKwon/SPARK-28321-doc.
Lead-authored-by: HyukjinKwon <gurwls223@apache.org>
Co-authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@HyukjinKwon
HyukjinKwon deleted the SPARK-28321 branch March 3, 2020 01:18
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viiryaviiryaviirya approved these changes
@srowensrowensrowen left review comments

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@HyukjinKwon@cloud-fan@SparkQA@gatorsmile@viirya@srowen@dongjoon-hyun