[R] SF columns in datasets with filters

[asfimport]

First reported at https://issues.apache.org/jira/browse/ARROW-10386?focusedCommentId=17331668&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17331668

OK, I actually have recreated a similar issue. In the following code, I create an sf object and write it as a dataset to parquet files. I then call open_dataset() on the files.

If I collect() the dataset I get back an sf object, no problem.

But if I first filter() the dataset then collect() I get an error.

library(sf)
library(arrow)
library(dplyr)

n <- 10000

fake <- tibble(
 ID=seq(n),
 Date=sample(seq(as.Date('2019-01-01'), as.Date('2021-04-01'), by=1), size=n, replace=TRUE),
 x=runif(n=n, min=-170, max=170),
 y=runif(n=n, min=-60, max=70),
 text1=sample(x=state.name, size=n, replace=TRUE),
 text2=sample(x=state.name, size=n, replace=TRUE),
 text3=sample(x=state.division, size=n, replace=TRUE),
 text4=sample(x=state.region, size=n, replace=TRUE),
 text5=sample(x=state.abb, size=n, replace=TRUE),
 num1=sample(x=state.center$x, size=n, replace=TRUE),
 num2=sample(x=state.center$y, size=n, replace=TRUE),
 num3=sample(x=state.area, size=n, replace=TRUE),
 Rand1=rnorm(n=n),
 Rand2=rnorm(n=n, mean=100, sd=3),
 Rand3=rbinom(n=n, size=10, prob=0.4)
)

# make it into an sf object
spat <- fake %>% 
 st_as_sf(coords=c('x', 'y'), remove=FALSE, crs = 4326)

class(spat)
class(spat$geometry)

# create new columns for partitioning and write to disk
spat %>% 
 mutate(Year=lubridate::year(Date), Month=lubridate::month(Date)) %>% 
 group_by(Year, Month) %>% 
 write_dataset('data/splits/', format='parquet')

spat_in <- open_dataset('data/splits/')

class(spat_in)

# it's an sf as expected
spat_in %>% collect() %>% class()
spat_in %>% collect() %>% pull(geometry) %>% class()

# it even plots
leaflet::leaflet() %>% 
 leaflet::addTiles() %>% 
 leafgl::addGlPoints(data=spat_in %>% collect())

# but if we filter first
spat_in %>% 
 filter(Year == 2020 & Month == 2) %>% 
 collect()

# we get this error
Error in st_geometry.sf(x) : 
 attr(obj, "sf_column") does not point to a geometry column.
Did you rename it, without setting st_geometry(obj) <- "newname"?
In addition: Warning message:
Invalid metadata$r 

Reporter: Jonathan Keane / @jonkeane
Assignee: Jonathan Keane / @jonkeane

Related issues:

• [R] Disable row-level metadata application on datasets (duplicates)

PRs and other links:

• GitHub Pull Request #10167

_{Note: This issue was originally created as ARROW-12542. Please see the migration documentation for further details.}

[asfimport]

Jonathan Keane / @jonkeane:
Ok I'm starting to see what's going on here: When collecting the whole dataset everything is fine. But when we filter, we have the filtered data but try to apply the whole list of metdata to that column, but there's a length mismatch, so the metadata application stops (that's what that warning is that you're seeing) and thus the SF column isn't getting reconstructed correctly.

This is the branch where that is being called, but we might want/need to filter the metadata higher https://github.com/apache/arrow/blob/master/r/R/metadata.R#L62-L65 I'm looking into this to see if we can fix it.

[asfimport]

Jonathan Keane / @jonkeane:
Oh, and interestingly, when writing the dataset it looks like when partitioning we're actually writing all of the metadata out to each partition (instead of partitioning it so that each partition only has the metadata for the rows in its partition)

> df <- tibble::tibble(
+ part = 1:10,
+ a = rep(1:2, 5),
+ x = rep(list(structure(1, foo = "bar"), structure(2, baz = "qux")), 5)
+ )
> 
> # write the dataset, and open it
> tmp <- tempfile()
> write_dataset(df, tmp, partitioning = "part", format = "parquet")
> ds <- open_dataset(tmp)
> 
> read_parquet(file.path(tmp, "part=1/part-0.parquet"))
# A tibble: 1 x 2
 a x
 <int> <list<double>>
1 1 [1]
Warning message:
Invalid metadata$r 
> 
> tab <- read_parquet(file.path(tmp, "part=1/part-0.parquet"), as_data_frame = FALSE)
> 
> tab$metadata$r
 'arrow_r_metadata' chr "A\n3\n262148\n197888\n5\nUTF-8\n531\n2\n531\n0\n1026\n1\n262153\n5\nnames\n16\n0\n254\n531\n3\n254\n254\n531\n2"| __truncated__
List of 2
 $ attributes: Named list()
 $ columns :List of 3
 ..$ part: NULL
 ..$ a : NULL
 ..$ x :List of 2
 .. ..$ attributes: NULL
 .. ..$ columns :List of 10
 .. .. ..$ :List of 2
 .. .. .. ..$ attributes:List of 1
 .. .. .. .. ..$ foo: chr "bar"
 .. .. .. ..$ columns : NULL
 .. .. ..$ :List of 2
 [...]

Solving that alone (probably) won't be enough to resolve the filter() issue as well, but will be part of the solution

[asfimport]

Neal Richardson / @nealrichardson:
If we're storing per-row metadata in the schema, that's not going to behave well. Perhaps we need to rethink how we store this in a way that round-trips better.