[R] Support creating ListArray from R list

[asfimport]

DF = data.frame(a = 1:10)
DF$b = as.list(DF$a)
arrow::write_parquet(DF, 'test.parquet')
# Error in Table__from_dots(dots, schema) : cannot infer type from data

This appears to be supported naturally already in Python:

import pandas as pd
pd.DataFrame({'a': [1, 2, 3], 'b': [[1, 2], [3, 4], [5, 6]]}).to_parquet('test.parquet')

Reporter: Michael Chirico
Assignee: Francois Saint-Jacques / @fsaintjacques

Related issues:

• [R] Refactor R <-> Array conversion (relates to)

PRs and other links:

• GitHub Pull Request #6275

_{Note: This issue was originally created as ARROW-7662. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
This logic would go in https://github.com/apache/arrow/blob/dd6b17d/r/src/array_from_vector.cpp#L871. I think you'd have to iterate over the list and call InferType on each element, and if they were all the same type, you could construct a ListType of that type (cf.

arrow/r/R/list.R

Line 30 in dd6b17d

list_of <- function(type) shared_ptr(ListType, list__(type))

). If they weren't all the same, there's probably a way to represent that in Arrow, but you could also just bail out like it does now; getting the uniform list type supported would probably solve most use cases.

Would you be interested in submitting a PR?

[asfimport]

Michael Chirico:
Hey @nealrichardson, I'm taking a look at this now (first time looking at src for arrow), it looks like it should be as simple as making a similar version as for StructType for ArrayType here:

https://github.com/apache/arrow/blob/master/r/src/array_from_vector.cpp#L861-L871

case VECSXP:
  if (Rf_inherits(x, "data.frame")) {
    R_xlen_t n = XLENGTH(x);
    SEXP names = Rf_getAttrib(x, R_NamesSymbol);
    std::vector<std::shared_ptr<arrow::Field>> fields(n);
    for (R_xlen_t i = 0; i < n; i++) {
      fields[i] = std::make_shared<arrow::Field>(CHAR(STRING_ELT(names, i)),
                                                 InferType(VECTOR_ELT(x, i)));
    }
    return std::make_shared<StructType>(std::move(fields));
  }

Does that sound about right?

[asfimport]

Michael Chirico:
I guess we were commenting at the same time, seems I found the right part of the code after all.

Is there a difference between ListType and StructType with respect to the need to enforce type consistency? My (rudimentary) understanding was that both StructType and ListType need fixed types – StructType as (KeyType, ValueType) and ListType as just ValueType. So I was planning to just imitate that logic.

[asfimport]

Neal Richardson / @nealrichardson:
Yes, there is a difference. Think of StructType as a data frame, so each row of the table has a row of the StructType in it.

> df <- data.frame(col1=1:3, col2=c("a", "b", "c"))
> a <- Array$create(df)
> a$type
StructType
struct<col1: int32, col2: dictionary<values=string, indices=int8, ordered=0>>
> length(a)
[1] 3

Each column in the data frame has a type, so each field of the StructType is a known type.

I think the tricky part is that for a data frame column, it is an R "list" underneath because that's what a data frame is, but the list elements are the columns of the struct, and the rows are the rows within each column of the data frame. But for list column, the list elements are the rows. So when we create list_of(), we need to know what type to provide there.

Just brainstorming, but I think you want something like (forgive the probably invalid code)

case VECSXP:
  if (Rf_inherits(x, "data.frame")) {
    (... the existing code)
  } else { // perhaps you want to check that the list is not named?
    R_xlen_t n = XLENGTH(x);
    if (n > 0) {
      std::shared_ptr<arrow::Type> element_type = InferType(VECTOR_ELT(x, 0);
      for (R_xlen_t i = 1; i < n; i++) {
        if (element_type != InferType(VECTOR_ELT(x, i)) {
          break;
        }
      }
      if (i == n) { // we made it through the loop and all are the same
        return std::make_shared<ListType>(element_type);
      }
  }

That said, I'm not sure how great the ListType support is currently in the R package. We might be able to infer the type but still have more left to implement.

> lcol <- list(1, 2, 3)
> Array$create(lcol)$type
Error in Array__from_vector(x, type) : cannot infer type from data
## That was expected, given that we haven't fixed that switch statement yet
## but what if we specify the type?
> Array$create(lcol, type = list_of(float64()))
Error in Array__from_vector(x, type) : 
  NotImplemented: type not implemented

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 6275
#6275