ARROW-5586: [R] convert Array of LIST type to R lists

[romainfrancois]

We don't have the reverse operation yet (convert from an R data structure) to a list array, so those aren't easy to make, but e.g. we can get some with the json reader:

library(arrow, warn.conflicts=FALSE)
tf<- tempfile()
writeLines(' { "hello": 3.5, "world": false, "yo": "thing", "arr": [1, 2, 3], "nuf": {} } { "hello": 3.25, "world": null, "arr": [2], "nuf": null } { "hello": 3.125, "world": null, "yo": "\u5fcd", "arr": [], "nuf": { "ps": 78 } } { "hello": 0.0, "world": true, "yo": null, "arr": null, "nuf": { "ps": 90 } }', tf)
tab1<- read_json_arrow(tf, as_tibble=FALSE)
list_array<-tab1$column(3L)$data()$chunk(0)
list_array#> arrow::ListArray #> [#> [#> 1,#> 2,#> 3#> ],#> [#> 2#> ],#> [],#> null#> ]list_array$values#> function() `arrow::Array`$dispatch(ListArray__values(self))#> <environment: 0x7f93c3b5cf88>list_array$value_length(0)
#> [1] 3list_array$value_offset(0)
#> [1] 0list_array$raw_value_offsets()
#> [1] 0 3 4 4list_array$as_vector()
#> [[1]]#> integer64#> [1] 1 2 3#> #> [[2]]#> integer64#> [1] 2#> #> [[3]]#> integer64#> character(0)#> #> [[4]]#> NULL

^{Created on 2019-06-14 by the reprex package (v0.3.0.9000)}

[fsaintjacques]

This is a relatively costly thing to do. I'm not sure how we can escape it. Do you keep a reference to the generated Slice/Array, or does it get consumed and transformed in something else. See #4366 (comment) for reference.

[romainfrancois]

::Allocate() makes an R list, and the various ::Ingest.*() fill that list.

I'll look if/how I can do this without actually calling Slice(), might be easy enough to just scan through values_array I guess.

[fsaintjacques]

The challenging part is dealing with the bitmap and null_count.

[fsaintjacques]

This can be a huge timesink, I think it's ok for now, but be wary that this will be a problem on large Array.

[romainfrancois]

I see. I don't think it's too much trouble to skip using Slice and use the bitmap of the values_array etc ... I'll have a look.

[romainfrancois]

🤔 yeah it's about the Array__as_vector(slice) which needs slice as a proper array already. The Converter api is designed to ingest all of an array, so I guess it would need some extra methods to only ingest a slice of an array without. I'll open another issue for that.

[romainfrancois]

https://issues.apache.org/jira/browse/ARROW-5628?filter=12344983

[fsaintjacques]

I find this file very error prone to read. I'd recommend going selective import route.

arrowExports.cpp # with ifdef import of impl/mock depending on ARROW_R_WITH_ARROW
arrowExports_impl.cpp
arrowExports_mock.cpp

Don't do this in this PR, open a followup ticket.

[romainfrancois]

Not that I would expect anyone to actually read that file (hence it is supposed to be hidden in pull requests, ...) but I've open that issue:

https://issues.apache.org/jira/browse/ARROW-5627

[fsaintjacques]

Is it worth adding support for FixedSizeList?

[romainfrancois]

this is new to me, that probably is another issue/pr.

[fsaintjacques]

yep, but the code will be similar, almost equal, FixedSizeList also export value_offset.

[fsaintjacques]

These are "low-level" functions, they don't perform any bound checking, you can segfault with the wrong i. Do they need to be exported?

[romainfrancois]

🤷‍♂ what's the rule about what is export worthy ?

[codecov-io]

Codecov Report

Merging #4575 into master will decrease coverage by 8.43%.
The diff coverage is 42.68%.

@@ Coverage Diff @@## master #4575 +/- ##
==========================================
- Coverage 83.42% 74.98% -8.44% 
==========================================
Files 209 56 -153 Lines 16062 3358 -12704 Branches 1253 0 -1253 ==========================================
- Hits 13399 2518 -10881 + Misses 2384 840 -1544 + Partials 279 0 -279

Impacted Files	Coverage Δ
r/R/array.R	63.63% <0%> (-11.37%)	⬇️
r/R/List.R	100% <100%> (ø)	⬆️
r/src/datatype.cpp	75.49% <100%> (+1%)	⬆️
r/src/arrowExports.cpp	72.46% <26.66%> (-1.44%)	⬇️
r/R/arrowExports.R	72.32% <28.57%> (-1.42%)	⬇️
r/src/array.cpp	64.4% <8.33%> (-14.76%)	⬇️
r/src/array__to_vector.cpp	76.88% <80.95%> (+0.24%)	⬆️
go/arrow/ipc/writer.go
go/arrow/math/uint64_amd64.go
go/arrow/memory/memory_avx2_amd64.go
... and 150 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0631357...300f897. Read the comment docs.