Skip to content

[SPARK-30278][SQL][DOC] Update Spark SQL document menu for new changes - #26917

Closed
xuanyuanking wants to merge 2 commits into
apache:masterfrom
xuanyuanking:SPARK-30278
Closed

[SPARK-30278][SQL][DOC] Update Spark SQL document menu for new changes#26917
xuanyuanking wants to merge 2 commits into
apache:masterfrom
xuanyuanking:SPARK-30278

Conversation

@xuanyuanking

@xuanyuankingxuanyuanking commented Dec 17, 2019

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Update the Spark SQL document menu and join strategy hints.

Why are the changes needed?

  • Several new changes in the Spark SQL document didn't change the menu-sql.yaml correspondingly.
  • Update the demo code for join strategy hints.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Document change only.

@xuanyuanking

xuanyuanking commented Dec 17, 2019

Copy link
Copy Markdown
MemberAuthor

cc @cloud-fan@gatorsmile

@SparkQA

SparkQA commented Dec 17, 2019

Copy link
Copy Markdown

Test build #115422 has finished for PR 26917 at commit 7222fa3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Dec 17, 2019
View reviewed changes
Comment threaddocs/sql-performance-tuning.md
{% highlight r %}
src <- sql("SELECT * FROM src")
records <- sql("SELECT * FROM records")
head(join(broadcast(src), records, src$key == records$key))

@cloud-fancloud-fanDec 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did we remove the broadcast method?

@xuanyuankingxuanyuankingDec 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the broadcast method still available:

spark/R/pkg/R/DataFrame.R

Lines 4129 to 4134 in 18431c7

setMethod("broadcast",
signature(x="SparkDataFrame"),
function(x) {
sdf<- callJStatic("org.apache.spark.sql.functions", "broadcast", x@sdf)
dataFrame(sdf)
})

@cloud-fancloud-fanDec 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what's the recommended API to add hint? df.hint(...) or hint(df, ...)?

@xuanyuankingxuanyuankingDec 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hint(df, ...) is for R, there's no df.hint(...) in R？

@cloud-fancloud-fanDec 18, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah nvm, didn't realize it's R code

@maropu

maropu commented Dec 17, 2019

Copy link
Copy Markdown
Member

How about adding the docs about the SQL case, too? #25464

@xuanyuanking

xuanyuanking commented Dec 17, 2019

Copy link
Copy Markdown
MemberAuthor

@maropu We have the SQL hint here: https://github.com/apache/spark/pull/26917/files#diff-ecc0a33250015b4a0fb4a5f1de5cc502R167, do you mean add demo for partition by hint?

@maropu

maropu commented Dec 17, 2019

Copy link
Copy Markdown
Member

do you mean add demo for partition by hint?

Yea, yes. we can accept REPARTITION_BY_RANGE and REPARTITION now.

@xuanyuanking

xuanyuanking commented Dec 18, 2019

Copy link
Copy Markdown
MemberAuthor

Sure, done in accf45c, please take a look.

@SparkQA

SparkQA commented Dec 18, 2019

Copy link
Copy Markdown

Test build #115516 has finished for PR 26917 at commit accf45c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Dec 27, 2019

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@xuanyuanking

xuanyuanking commented Dec 27, 2019

Copy link
Copy Markdown
MemberAuthor

Thanks!

@xuanyuanking
xuanyuanking deleted the SPARK-30278 branch December 27, 2019 05:24
@maropu

maropu commented Dec 28, 2019

Copy link
Copy Markdown
Member

Thanks for the update, @xuanyuanking !

fqaiser94 pushed a commit to fqaiser94/spark that referenced this pull request Mar 30, 2020
@xuanyuanking@fqaiser94
[SPARK-30278][SQL][DOC] Update Spark SQL document menu for new changes
84918e7
### What changes were proposed in this pull request?
Update the Spark SQL document menu and join strategy hints.
### Why are the changes needed?
- Several new changes in the Spark SQL document didn't change the menu-sql.yaml correspondingly.
- Update the demo code for join strategy hints.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Document change only.
Closesapache#26917 from xuanyuanking/SPARK-30278.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fancloud-fancloud-fan left review comments

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xuanyuanking@SparkQA@maropu@cloud-fan@dongjoon-hyun