Skip to content

ARROW-15622: [R] Implement union_all and union for arrow_dplyr_query - #13090

Closed
wjones127 wants to merge 7 commits into
apache:masterfrom
wjones127:ARROW-15622-union-all
Closed

ARROW-15622: [R] Implement union_all and union for arrow_dplyr_query#13090
wjones127 wants to merge 7 commits into
apache:masterfrom
wjones127:ARROW-15622-union-all

Conversation

@wjones127

@wjones127wjones127 commented May 6, 2022

Copy link
Copy Markdown
Member

This PR adds support for dplyr::union and dplyr::union_all. Not sure why, but I find I must use the fully qualified name dplyr::union or else will get an error.

@github-actions

Copy link
Copy Markdown

@wjones127wjones127 changed the title ARROW-15622: Implement union_all and union for arrow_dplyr_queryARROW-15622: [R] Implement union_all and union for arrow_dplyr_queryMay 9, 2022
@wjones127

Copy link
Copy Markdown
MemberAuthor

Some example usage:

library(arrow)
#> #> Attaching package: 'arrow'#> The following object is masked from 'package:utils':#> #> timestamp
library(dplyr)
#> #> Attaching package: 'dplyr'#> The following objects are masked from 'package:stats':#> #> filter, lag#> The following objects are masked from 'package:base':#> #> intersect, setdiff, setequal, uniontab1<- arrow_table(x=1:3)
tab2<- arrow_table(x=2:4, y= c("a", "b", "c"))
tab1|>
mutate(y="a") |>
union_all(tab2) |>
collect()
#> # A tibble: 6 × 2#> x y #> <int> <chr>#> 1 2 a #> 2 3 b #> 3 4 c #> 4 1 a #> 5 2 a #> 6 3 atab1|>
mutate(y="a") |>
union_all(tab2) |>
arrange(x, y) |>
collect()
#> # A tibble: 6 × 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 a #> 3 2 a #> 4 3 a #> 5 3 b #> 6 4 ctab1|>
mutate(y="a") |>dplyr::union(tab2) |>
arrange(x, y) |>
collect()
#> # A tibble: 5 × 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 a #> 3 3 a #> 4 3 b #> 5 4 c

Created on 2022-05-09 by the reprex package (v2.0.1)

@wjones127
wjones127force-pushed the ARROW-15622-union-all branch from fb0b93a to 2d6d125CompareMay 9, 2022 20:34
@wjones127
wjones127 marked this pull request as ready for review May 9, 2022 21:31
@wjones127
wjones127force-pushed the ARROW-15622-union-all branch from 2d6d125 to 1640771CompareMay 23, 2022 16:26
@wjones127

Copy link
Copy Markdown
MemberAuthor

The timeouts on RTools mingw seem to be random. I ran locally on mingw64 and was able to get it to build and pass all R tests.

@nealrichardson

Copy link
Copy Markdown
Member

Not sure why, but I find I must use the fully qualified name dplyr::union or else will get an error.

Probably namespace collision:

> union
function (x, y) {
u <- as.vector(x)
v <- as.vector(y)
unique(c(u, v))
}
<bytecode: 0x1290ae500>
<environment: namespace:base>

@nealrichardsonnealrichardson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of suggestions on the tests but otherwise LGTM, thanks!

Comment threadr/tests/testthat/test-dplyr-union.R Outdated
Comment threadr/tests/testthat/test-dplyr-union.R Outdated
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
@ursabot

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 6576aa0 and contender = d889ade. d889ade is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.7% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.37% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.47% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] d889adec ec2-t3-xlarge-us-east-2
[Failed] d889adec test-mac-arm
[Failed] d889adec ursa-i9-9960x
[Finished] d889adec ursa-thinkcentre-m75q
[Finished] 6576aa06 ec2-t3-xlarge-us-east-2
[Failed] 6576aa06 test-mac-arm
[Failed] 6576aa06 ursa-i9-9960x
[Finished] 6576aa06 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot

Copy link
Copy Markdown

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm

kou pushed a commit that referenced this pull request Feb 20, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@wjones127@nealrichardson@ursabot