GH-35692: [C++][Parquet] Support to read fixed size list array with nulls by HuaHuaY · Pull Request #50271 · apache/arrow

HuaHuaY · 2026-06-27T07:14:26Z

Rationale for this change

Support to read fixed size list array with nulls.

What changes are included in this PR?

Modified FixedSizeListReader to add null slots to FixedSizeList.

Are these changes tested?

Yes.

Are there any user-facing changes?

User can read fixed size list array with nulls now.

This PR also fixesGH-35697.

GitHub Issue: [C++][Parquet] Null fixed-length lists cannot be read from parquet file #35692

pitrou · 2026-07-08T09:09:28Z

+ int32_t expected_size) -> Status {
+ std::span<const int32_t> run_offsets(offsets + start,
+ static_cast<size_t>(length + 1));
+ const auto first_invalid_offset = std::ranges::adjacent_find(


I'm not sure C++20 ranges are available on all our CI platforms unfortunately (some of them require old compilers).

It looks like there are no compilation errors in the CI pipelines which are already run.

Let me enable the more annoying ones :)

The new CI tests have also been compiled, so I haven't changed the code yet.

@thisisnic Do you remember how to launch the R builds with the most outdated compilers? :)

Not off the top of my head, but there should be a recent-ish PR which fixed the last ones that broke it, and will have the CI job being run on it in the comments.

pitrou · 2026-07-08T09:15:21Z

@rok FYI :)

wgtmac · 2026-07-21T09:44:15Z

+ return Status::OK();
+ }
+ child_arrays.push_back(
+ ::arrow::MakeArray(data->child_data[0]->Slice(offsets[start], child_length)));


Could we avoid creating one Array per validity run? This could become expensive for wide fixed-size lists. Is it possible to use a single builder using AppendArraySlice() and AppendNulls() instead?

I don't think this is a good idea. First, the array is created via Slice, and the actual data isn't copied. Second, Concatenate handles different array types automatically, whereas doing it manually would be too much of a hassle.

My concern is the per-run object/allocation overhead rather than copying each valid slice: an alternating validity bitmap still creates O(number of runs) Array/ArrayData objects and O(number of null runs) null arrays before the final concatenation.
But I agree that my suggestion requires a lot of changes to handle different child types. I'm fine to keep it as-is by adding a TODO comment for a future improvement?

I have added a TODO comment in the latest commit.

wgtmac · 2026-07-21T09:54:22Z

Codex reminded me that code below is outdated after this fix and it seems that level_info.null_slot_usage is useless anymore.

arrow/cpp/src/parquet/level_conversion.cc

Lines 122 to 126 in 0077539

if (output->null_count > 0 && level_info.null_slot_usage > 1) { 
throwParquetException( 
"Null values with null_slot_usage > 1 not supported."
"(i.e. FixedSizeLists with null values are not supported)"); 
 } 

HuaHuaY · 2026-07-21T11:01:56Z

Codex reminded me that code below is outdated after this fix and it seems that level_info.null_slot_usage is useless anymore.

This requires modifying the public methods in cpp/src/parquet/level_conversion.h, and I prefer opening a new pull request to address it.

pitrou · 2026-07-22T13:48:46Z

@HuaHuaY Are you waiting for another review here?

HuaHuaY · 2026-07-22T15:37:57Z

@HuaHuaY Are you waiting for another review here?

If there are no new comments, I think this PR is ok to be merged.

wgtmac

+1

wgtmac · 2026-07-27T02:40:49Z

I'll merge this tomorrow if no objection.

thisisnic · 2026-07-27T15:19:37Z

@github-actions crossbow submit -g r

thisisnic · 2026-07-27T15:20:35Z

I'll merge this tomorrow if no objection.

I've triggered the R builds just to make sure we don't have potential failures suggested in #50271 (comment)

HuaHuaY · 2026-07-27T15:23:01Z

I'll merge this tomorrow if no objection.
I've triggered the R builds just to make sure we don't have potential failures suggested in #50271 (comment)

It seems that no extra ci starts. I might be because @pitrou has added "CI: Extra R". R's CI has already been run.

github-actions · 2026-07-27T15:25:44Z

Revision: 06266f9

Submitted crossbow builds: ursacomputing/crossbow @ actions-d711fbc610

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions
test-r-wasm

HuaHuaY · 2026-07-27T15:26:41Z

Oh. Extra CI starts. It just takes a little while to start up.

HuaHuaY · 2026-07-27T15:33:26Z

The additional CI run revealed an issue. Arrow recently introduced simdjson but hasn't updated this CI configuration yet. https://github.com/ursacomputing/crossbow/actions/runs/30279773748/job/90022897470

HuaHuaY · 2026-07-28T02:46:15Z

I think these failures are unrelated to this PR. To be sure, I'll rebase and run it again.

HuaHuaY · 2026-07-28T02:48:01Z

@github-actions crossbow submit -g r

github-actions · 2026-07-28T02:48:37Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/30324136965

wgtmac · 2026-07-28T02:53:31Z

@github-actions crossbow submit -g r

github-actions · 2026-07-28T02:56:24Z

Revision: 6b0bd58

Submitted crossbow builds: ursacomputing/crossbow @ actions-523da37569

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions
test-r-wasm

wgtmac · 2026-07-28T03:54:58Z

I've checked that all CI failures are unrelated.

conbench-apache-arrow · 2026-07-28T10:57:19Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 5e698fe.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

) ### Rationale for this change Fix#50636 - `test-r-macos-as-cran` nightly job fails compiling `visit({range_start, range_cur})` introduced in #50248. ```console /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/compute/kernels/vector_sort.cc:325:11: note: candidate function not viable: cannot convert initializer list argument to 'std::span<uint64_t>' (aka 'span<unsigned long long>') 325 | [&](std::span<uint64_t> indices) { SortNextColumn(indices, offset); }); ``` The job pins [macOS SDK 11.3](https://github.com/ursacomputing/crossbow/actions/runs/30420027426/job/90474737457#step:9:14), so libc++ there does not have C++20 iterator-pair span constructor available yet ([available with libc++ 14](https://libcxx.llvm.org/Status/Cxx20.html)). Similar problem as in recent #50295 ### What changes are included in this PR? Replace std::span iterator-pair constructor with subspan in `vector_sort.cc` Also replace std::ranges in `parquet/arrow/reader.cc` introduced in #50271 ### Are these changes tested? Yes, builds locally and crossbow `test-r-macos-as-cran` job succeeds. ### Are there any user-facing changes? No. * GitHub Issue: #50636 Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

HuaHuaY requested review from pitrou and wgtmac as code owners June 27, 2026 07:14

github-actionsBot added Component: Parquet Component: C++ awaiting review Awaiting review labels Jun 27, 2026

pitrou reviewed Jul 8, 2026
View reviewed changes

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 8, 2026

pitrou added the CI: Extra: R Run extra R CI label Jul 8, 2026

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 915c757 to 2fc0d49Compare July 9, 2026 09:42

github-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 13, 2026

wgtmac reviewed Jul 21, 2026
View reviewed changes

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 2fc0d49 to c5cadcfCompare July 21, 2026 11:00

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 21, 2026

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from c5cadcf to 3840ffcCompare July 22, 2026 16:09

wgtmac approved these changes Jul 22, 2026
View reviewed changes

Comment threadcpp/src/parquet/arrow/arrow_reader_writer_test.cc Outdated

wgtmac approved these changes Jul 24, 2026
View reviewed changes

HuaHuaY added 5 commits July 28, 2026 10:47

support to read fixed size list array with nulls
8338513

address review
fb4a9f7

address review
bbecb76

address review
f8cfd46

address review
6b0bd58

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 06266f9 to 6b0bd58Compare July 28, 2026 02:47

wgtmac merged commit 5e698fe into apache:mainJul 28, 2026
57 of 61 checks passed

wgtmac removed the awaiting change review Awaiting change review label Jul 28, 2026

wgtmac mentioned this pull request Jul 28, 2026
[C++][Parquet] Null fixed-length lists cannot be read from parquet file #35692
Closed

HuaHuaY deleted the nested_fixed_size_list_reader branch July 28, 2026 03:56

HuaHuaY mentioned this pull request Jul 28, 2026
[C++][Parquet] Remove unused member null_slot_usage in struct LevelInfo#50678
Closed

tadeja mentioned this pull request Jul 29, 2026
GH-50636: [C++] Replace std::span/ranges usage to fix macOS CRAN #50705
Merged

pitrou mentioned this pull request Jul 29, 2026
[C++][CI] Valgrind errors in parquet-arrow-reader-writer-test#50718
Closed

lxy-9602 mentioned this pull request Aug 20, 2026
fix(parquet): support nullable fixed-size lists for vector apache/paimon-cpp#231
Merged

Uh oh!

Conversation

HuaHuaY commented Jun 27, 2026• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaYJul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jul 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaYJul 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Jul 21, 2026

Uh oh!

HuaHuaY commented Jul 21, 2026

Uh oh!

pitrou commented Jul 22, 2026

Uh oh!

HuaHuaY commented Jul 22, 2026

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Jul 27, 2026

Uh oh!

thisisnic commented Jul 27, 2026

Uh oh!

thisisnic commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuaHuaY commented Jul 28, 2026

Uh oh!

HuaHuaY commented Jul 28, 2026

Uh oh!

github-actionsBot commented Jul 28, 2026

Uh oh!

wgtmac commented Jul 28, 2026

Uh oh!

github-actionsBot commented Jul 28, 2026

Uh oh!

wgtmac commented Jul 28, 2026

Uh oh!

Uh oh!

HuaHuaY commented Jun 27, 2026•
edited by github-actions Bot
Loading

HuaHuaYJul 9, 2026•
edited
Loading

HuaHuaYJul 22, 2026•
edited
Loading

HuaHuaY commented Jul 27, 2026•
edited
Loading

HuaHuaY commented Jul 27, 2026•
edited
Loading