[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in R by dilipbiswal · Pull Request #22107 · apache/spark

dilipbiswal · 2018-08-14T18:14:59Z

What changes were proposed in this pull request?

SPARK-21274 added support for EXCEPT ALL and INTERSECT ALL. This PR adds the support in R.

How was this patch tested?

Added test in test_sparkSQL.R

SparkQA · 2018-08-14T19:16:25Z

Test build #94760 has finished for PR 22107 at commit 426ffed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-08-14T19:24:05Z

cc @felixcheung

SparkQA · 2018-08-14T19:52:55Z

Test build #94764 has finished for PR 22107 at commit 7e88c9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheungu · 2018-08-15T00:40:33Z

+#' intersectAllDF <- intersectAll(df1, df2)
+#' }
+#' @rdname intersectAll
+#' @note intersectAll since 2.4


please put 2.4.0

@felixcheungu Ok.

SparkQA · 2018-08-15T01:54:19Z

Test build #94777 has finished for PR 22107 at commit 5247ab5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-08-15T02:27:39Z

+ list("a", 1),
+ list("a", 1),
+ list("b", 3),
+ list("c", 4)),


nit:
list(list("a", 1), list("a", 1), list("a", 1), list("a", 1), list("b", 3), list("c", 4)),

HyukjinKwon · 2018-08-15T02:30:19Z

+ schema = c("a", "b"))
+ df2 <- createDataFrame(
+ list(list("a", 1), list("a", 1), list("b", 3)),
+ schema = c("a", "b"))


nit:
df2<- createDataFrame(list(list("a", 1), list("a", 1), list("b", 3)), schema= c("a", "b"))

HyukjinKwon · 2018-08-15T02:32:06Z

+ stringsAsFactors = FALSE)
+ except_all_expected <- data.frame("a" = c("a", "a", "c"), "b" = c(1, 1, 4),
+ stringsAsFactors = FALSE)
+ intersect_all_df <- arrange(intersectAll(df1, df2), df1$a)


Strictly, the naming rule is intersectAllDf or intersect.all.df (see #17590 (comment))

HyukjinKwon · 2018-08-15T02:33:24Z

Seems fine.

SparkQA · 2018-08-15T07:05:01Z

Test build #94789 has finished for PR 22107 at commit 1d93304.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-08-15T07:06:57Z

retest this please

SparkQA · 2018-08-15T07:51:37Z

Test build #94791 has finished for PR 22107 at commit 1d93304.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2018-08-16T06:30:30Z

+#' df2 <- read.json(path2)
+#' exceptAllDF <- exceptAll(df1, df2)
+#' }
+#' @rdname exceptAll


this is a bug in except there should only be one @rdname for each

@felixcheung Thanks .. Did you want the original function except fixed at part of this ?

felixcheung · 2018-08-16T06:31:19Z

+#' df2 <- read.json(path2)
+#' intersectAllDF <- intersectAll(df1, df2)
+#' }
+#' @rdname intersectAll


felixcheung · 2018-08-16T06:31:57Z

+ function(x, y) {
+ intersected <- callJMethod(x@sdf, "intersectAll", y@sdf)
+ dataFrame(intersected)
+ })


add extra empty line after code

@felixcheung OK.

felixcheung · 2018-08-16T06:32:14Z

+ excepted <- callJMethod(x@sdf, "exceptAll", y@sdf)
+ dataFrame(excepted)
+ })
+


nit: remove one of the two empty lines

@felixcheung Sure.

SparkQA · 2018-08-16T08:16:33Z

Test build #94844 has finished for PR 22107 at commit 528050d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2018-08-16T20:14:18Z

@felixcheung I have incorporated the comments.

felixcheung

LGTM

HyukjinKwon

LGTM too

felixcheung · 2018-08-17T07:13:52Z

merged to master

dilipbiswal · 2018-08-17T07:25:00Z

Thank you very much @HyukjinKwon @felixcheung

dilipbiswal added 3 commits August 14, 2018 11:06

[SPARK-25117] Add EXEPT ALL and INTERSECT ALL support in R
6076700

minor fix
426ffed

lint
7e88c9d

dilipbiswal changed the title ~~[SPARK-25117] Add EXEPT ALL and INTERSECT ALL support in R~~[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in RAug 14, 2018

felixcheungu reviewed Aug 15, 2018
View reviewed changes

code review
5247ab5

HyukjinKwon reviewed Aug 15, 2018
View reviewed changes

code review
1d93304

felixcheung reviewed Aug 16, 2018
View reviewed changes

code review
528050d

felixcheung approved these changes Aug 17, 2018
View reviewed changes

HyukjinKwon approved these changes Aug 17, 2018
View reviewed changes

asfgit closed this in 162326cAug 17, 2018

Uh oh!

Conversation

dilipbiswal commented Aug 14, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 14, 2018

Uh oh!

gatorsmile commented Aug 14, 2018

Uh oh!

SparkQA commented Aug 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Aug 15, 2018

Uh oh!

SparkQA commented Aug 15, 2018

Uh oh!

HyukjinKwon commented Aug 15, 2018

Uh oh!

SparkQA commented Aug 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 16, 2018

Uh oh!

dilipbiswal commented Aug 16, 2018

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Aug 17, 2018

Uh oh!

dilipbiswal commented Aug 17, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants