Uh oh!
There was an error while loading. Please reload this page.
[SPARK-25328][PYTHON] Add an example for having two columns as the grouping key in group aggregate pandas UDF - #22329
[SPARK-25328][PYTHON] Add an example for having two columns as the grouping key in group aggregate pandas UDF#22329HyukjinKwon wants to merge 3 commits into
Conversation
SparkQA
commented
Sep 4, 2018
Test build #95666 has finished for PR 22329 at commit
|
| | 1|1.5| | ||
| | 2|6.0| | ||
| +---+---+ | ||
| >>> @pandas_udf("id long, v1 double, v2 double", PandasUDFType.GROUPED_MAP) # doctest: +SKIP |
There was a problem hiding this comment.
It took me a while to realize v1 is a grouping key. It also a bit uncommon to use double value as a grouping key . How about we do sth like?
id long, additional_key long, v double
SparkQA
commented
Sep 5, 2018
Test build #95690 has finished for PR 22329 at commit
|
HyukjinKwon
commented
Sep 5, 2018
cc @gatorsmile and @BryanCutler |
| | 2|6.0| | ||
| +---+---+ | ||
| >>> @pandas_udf( | ||
| ... "id long, additional_key double, v double", |
There was a problem hiding this comment.
do you mind changing the type of additional_key to long? It seems like the type coercion here is not necessary.
There was a problem hiding this comment.
Sorry, I know you just changed it, but I think just naming the column "ceil(v1 / 2)" with a type long would be a little more clear. Although "additional_key" is ok too, if you guys want to keep that.
SparkQA
commented
Sep 6, 2018
Test build #95734 has finished for PR 22329 at commit
|
icexelloss
commented
Sep 6, 2018
LGTM |
BryanCutler
commented
Sep 6, 2018
merged to master, thanks @HyukjinKwon . I just saw branch-2.4 was cut already, I'll see if I can figure out how to merge there too. |
…ouping key in group aggregate pandas UDF ## What changes were proposed in this pull request? This PR proposes to add another example for multiple grouping key in group aggregate pandas UDF since this feature could make users still confused. ## How was this patch tested? Manually tested and documentation built. Closes#22329 from HyukjinKwon/SPARK-25328. Authored-by: hyukjinkwon <gurwls223@apache.org> Signed-off-by: Bryan Cutler <cutlerb@gmail.com> (cherry picked from commit 7ef6d1d) Signed-off-by: Bryan Cutler <cutlerb@gmail.com>
BryanCutler
commented
Sep 6, 2018
merged to branch-2.4 |
HyukjinKwon
commented
Sep 7, 2018
Thanks guys :-) |
What changes were proposed in this pull request?
This PR proposes to add another example for multiple grouping key in group aggregate pandas UDF since this feature could make users still confused.
How was this patch tested?
Manually tested and documentation built.