ARROW-3814: [R] RecordBatch$from_arrays()

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[romainfrancois]

This started out as an implementation of RecordBatch$from_arrays() (i.e. https://issues.apache.org/jira/browse/ARROW-3814?filter=12344983) but now looks more like this issue: https://issues.apache.org/jira/browse/ARROW-3815?filter=12344983

The idea being that the record batch factory record_batch() would work with ... and schema, where each thing in the ... could be:

• an arrow::Array
• an R vector that can be converted to an array using array()

So where we had this before:

record_batch(tibble::tibble(x=1:10, y=1:10))

we would now have:

record_batch(x=1:10, y=1:10)

We would still be able to start from a data frame, via splicing, e.g.:

tbl <- tibble::tibble(x = 1:10, y = 1:10)
record_batch(!!!tbl)

So there would be no need for a RecordBatch$fromArray() method.

[romainfrancois]

I suppose the situation is similar for the table() factory

[codecov-io]

Codecov Report

Merging #3565 into master will decrease coverage by 11.02%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247

Impacted Files	Coverage Δ
src/table.cpp	64.17% <0%> (-4.25%)	⬇️
src/array_from_vector.cpp	78.29% <0%> (-0.05%)	⬇️
R/write_arrow.R	96.29% <0%> (ø)	⬆️
R/feather.R	58.33% <0%> (ø)	⬆️
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
cpp/src/plasma/client.cc
cpp/src/arrow/io/test-common.h
... and 685 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5025126...ab0cd16. Read the comment docs.

[romainfrancois]

proposal for the table() function, that might be renamed Table() perhaps. The idea is that the function can handle two cases:

• a variable list of record batches, which then results to a call to arrow::Table::FromRecordBatches :

library(arrow, warn.conflicts=FALSE)
library(purrr)
batch<- record_batch(x=1:2, y=letters[1:2])
# variable number of batchestab<- table(batch, batch, batch)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 6 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 1 a #> 4 2 b #> 5 1 a #> 6 2 b# splicing supportbatches<- map(1:10, ~record_batch(x=., y=letters[.]))
tab<- table(!!!batches)
tab#> arrow::Table
as_tibble(tab)
#> # A tibble: 10 x 2#> x y #> <int> <chr>#> 1 1 a #> 2 2 b #> 3 3 c #> 4 4 d #> 5 5 e #> 6 6 f #> 7 7 g #> 8 8 h #> 9 9 i #> 10 10 j

• a named list of R vectors, R arrays or chunked arrays, e.g.

library(arrow, warn.conflicts=FALSE)
a<-array(rnorm(10))
tab<- table(x=1:10, y=letters[1:10], z=a)
tab$schema#> arrow::Schema #> x: int32#> y: string#> z: double
as_tibble(tab)
#> # A tibble: 10 x 3#> x y z#> <int> <chr> <dbl>#> 1 1 a 1.68 #> 2 2 b 1.61 #> 3 3 c 0.879#> 4 4 d 0.315#> 5 5 e 0.877#> 6 6 f -1.28 #> 7 7 g 0.827#> 8 8 h 0.494#> 9 9 i 1.60 #> 10 10 j -1.66# supports splicing too, e.g. tab<- table(
row.number=1:150, # R integer vector -> converted to an Array!!!iris, # columns of iris are spliced, each is converted to an arrayarr=array(rnorm(150)) # an Array already
)
tab$schema#> arrow::Schema #> row.number: int32#> Sepal.Length: double#> Sepal.Width: double#> Petal.Length: double#> Petal.Width: double#> Species: dictionary<values=string, indices=int8, ordered=0>#> arr: double
as_tibble(tab)
#> # A tibble: 150 x 7#> row.number Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> <int> <dbl> <dbl> <dbl> <dbl> <fct> #> 1 1 5.1 3.5 1.4 0.2 setosa #> 2 2 4.9 3 1.4 0.2 setosa #> 3 3 4.7 3.2 1.3 0.2 setosa #> 4 4 4.6 3.1 1.5 0.2 setosa #> 5 5 5 3.6 1.4 0.2 setosa #> 6 6 5.4 3.9 1.7 0.4 setosa #> 7 7 4.6 3.4 1.4 0.3 setosa #> 8 8 5 3.4 1.5 0.2 setosa #> 9 9 4.4 2.9 1.4 0.2 setosa #> 10 10 4.9 3.1 1.5 0.1 setosa #> # … with 140 more rows, and 1 more variable: arr <dbl>

[xhochy]

Rebased.

[romainfrancois]

@xhochy was there a special need to rebase ? Just curious.
I usually only need to rebase after an R specific PR is squashed

[xhochy]

I rebased master on the release commit of the JavaScript 0.4.0 release.

We sadly only add the tagging commits to master once the release vote has passed. Then we need to rebase all PRs as we keep merging patches while the release vote runs.

[romainfrancois]

I think this is ready now.

[wesm]

Minor comments but this looks like a nice improvement. I think this can be merged after some small fixes /c comments

[romainfrancois]

Thanks. I’ll deal with those in the morning.

[wesm]

@romainfrancois this needs to be rebased now after the lint fixes, sorry about that

[romainfrancois]

Did we need to pass $1 to run_cpp_lint.py too ?

Removing it here allows to use

./r/lint.sh --fix

and let the tool fix the format

[wesm]

I think this is okay

[romainfrancois]

I think this is good to go, but I'd like to merge #4413 first because currently this pr would fail against the new Dictionary changes from #4316

[wesm]

Sounds good

[wesm]

+1

[wesm]

I think this is okay

[wesm]

If schema were a column name I guess you would have to pass the arguments differently

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.92%. Comparing base (5025126) to head (ab0cd16).

❗ There is a different number of reports uploaded between BASE (5025126) and HEAD (ab0cd16). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5025126)	HEAD (ab0cd16)
	5	1

Additional details and impacted files

@@ Coverage Diff @@## master #3565 +/- ##
===========================================
- Coverage 87.94% 76.92% -11.03% 
===========================================
Files 737 51 -686 Lines 81709 1976 -79733 Branches 1253 0 -1253 ===========================================
- Hits 71863 1520 -70343 + Misses 9599 456 -9143 + Partials 247 0 -247 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.