[R] Parquets do not support list-columns of ordered factors (ordered dictionaries)

[lgaborini]

Describe the bug, including details regarding any error messages, version, and platform.

With arrow 23.0.1.2:

data <- tibble::tibble(
 x = list(
 factor(c("a", "b"), levels = c("c", "b", "a"), ordered = TRUE),
 factor(c("b", "c"), levels = c("c", "b", "a"), ordered = TRUE)
 )
)
arrow::write_parquet(data, tempfile())
#> Error: Invalid: Column data for field 0 with type list<item: dictionary<values=string, indices=int8, ordered=0>> is inconsistent with schema list<item: dictionary<values=string, indices=int8, ordered=1>>

No issues if x is a list-column of factors (coerced to unsorted dictionaries).
Upstream bug of ropensci/targets#1291.

Component(s)

R

[aviralgarg05]

Hi I would like to solve this issue if someone could assign me

[thisisnic]

@aviralgarg05 - currently this is an untriaged bug ticket. Before we assign it to anyone, we need to work out if it can be replicated, if it's a bug or not (did this work in previous versions of Arrow - is it a bug or an unimplemented feature?), and then, if it is a bug, work out why it occurs.

Feel free to post your hypothesis about the cause here, and we can walk through your ideas.

[lgaborini]

Hello, it was probably an unimplemented or never properly tested feature.
I encountered this first in 2024 with arrow 16.1.0 (forgot to submit the bug report), then never actually checked again.

[thisisnic]

I did a little investigation with Claude, and it looks a bit more complex than it first appeared; will paste the output at the bottom of this comment shortly.

---

Root cause: the C++ DictionaryBuilder doesn't track the ordered flag.

When R converts an ordered factor to an Arrow array, the type is correctly inferred as dictionary<values=string, indices=int8, ordered=1>. However, the C++ DictionaryBuilder reconstructs the dictionary type on Finish() using:

return ::arrow::dictionary(indices_builder_.type(), value_type_);

The ordered parameter defaults to false, so the built array always has ordered=0 regardless of the input type.

For top-level ordered factor columns, there's already a workaround in the R package — RDictionaryConverter::ToChunkedArray() detects the mismatch and patches the type back to ordered=true.

However, when an ordered factor is inside a list column, ListBuilder::Finish() internally finishes its child DictionaryBuilder, producing ordered=0. The RDictionaryConverter::ToChunkedArray() workaround is never called for child converters — only the list converter's ToChunkedArray() runs. So the schema says list<dictionary<..., ordered=1>> but the data has list<dictionary<..., ordered=0>>, and validation catches the mismatch.

This has likely been broken since ordered dictionary support was added — as the reporter notes, it reproduces as far back as Arrow 16.1.0.

Proposed fix:

The fix belongs in C++, since adding another R-side workaround would be fragile (the ListBuilder also reconstructs its own type from the child builder's type via ListBuilder::type(), so both the list type and child type end up wrong).

We should open a separate C++ ticket for this, but here's the full picture of what needs to change:

1. MakeDictionaryBuilder currently decomposes the DictionaryType into separate index_type and value_type before passing them to the builder via DictionaryBuilderCase. The ordered flag is lost at this point. It should pass the full DictionaryType (or at minimum, the ordered flag) through to the builder.

2. DictionaryBuilderBase stores value_type_ but has no ordered_ member. It needs to store the ordered flag (or the full dictionary type).

3. DictionaryBuilderBase::type() currently returns ::arrow::dictionary(indices_builder_.type(), value_type_) which defaults ordered to false. It needs to pass the stored flag: ::arrow::dictionary(indices_builder_.type(), value_type_, ordered_).

4. DictionaryBuilderBase::FinishInternal() sets the output type via (*out)->type = type(), so fixing type() would fix Finish() automatically.

5. Once the C++ fix is in place, the R workaround and its TODO can be removed.

Note: DictionaryBuilderBase has multiple constructor overloads that would all need to accept the flag (or a default of false for backward compatibility).

[thisisnic]

Issue resolved by pull request 49937
#49937