Skip to content

[SPARK-11946][SQL] Audit pivot API for 1.6. - #9929

Closed
rxin wants to merge 2 commits into
apache:masterfrom
rxin:SPARK-11946
Closed

[SPARK-11946][SQL] Audit pivot API for 1.6.#9929
rxin wants to merge 2 commits into
apache:masterfrom
rxin:SPARK-11946

Conversation

@rxin

@rxinrxin commented Nov 24, 2015

Copy link
Copy Markdown
Contributor

Currently pivot's signature looks like

@scala.annotation.varargs
defpivot(pivotColumn: Column, values: Column*):GroupedData@scala.annotation.varargs
defpivot(pivotColumn: String, values: Any*):GroupedData

I think we can remove the one that takes "Column" types, since callers should always be passing in literals. It'd also be more clear if the values are not varargs, but rather Seq or java.util.List.

I also made similar changes for Python.

@rxin

rxin commented Nov 24, 2015

Copy link
Copy Markdown
ContributorAuthor

cc @aray

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aray do you know why we have a "sort" in here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sort is there to ensure that the output columns are in a consistent logical order.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok thanks - i'm going to add a comment there to explain.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done as part of #9603 (comment) but it is way too small to deserve its own pr.

@SparkQA

Copy link
Copy Markdown

Test build #46591 has finished for PR 9929 at commit 73d37fc.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use _to_java_column(pivot_col) and _to_seq() here? or df.pivot(df.a) may fail

@SparkQA

Copy link
Copy Markdown

Test build #2104 has finished for PR 9929 at commit 73d37fc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@davies

Copy link
Copy Markdown
Contributor

LGTM

@rxin

rxin commented Nov 24, 2015

Copy link
Copy Markdown
ContributorAuthor

Thanks - merging this in.

asfgit pushed a commit that referenced this pull request Nov 24, 2015
Currently pivot's signature looks like
```scala
scala.annotation.varargs
def pivot(pivotColumn: Column, values: Column*): GroupedData
scala.annotation.varargs
def pivot(pivotColumn: String, values: Any*): GroupedData
```
I think we can remove the one that takes "Column" types, since callers should always be passing in literals. It'd also be more clear if the values are not varargs, but rather Seq or java.util.List.
I also made similar changes for Python.
Author: Reynold Xin <rxin@databricks.com>
Closes#9929 from rxin/SPARK-11946.
(cherry picked from commit f315272)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@rxin@SparkQA@davies@aray