Uh oh!
There was an error while loading. Please reload this page.
GH-40171: [Python] Add Type_FIXED_SIZE_LIST to _NESTED_TYPES set - #40172
Merged
Conversation
jorisvandenbossche
approved these changes
Feb 27, 2024
jorisvandenbossche
left a comment
Member
There was a problem hiding this comment.
Thanks! This looks like an oversight in is_nested indeed
jorisvandenbossche
commented
Feb 27, 2024
Member
@danepitkin too late for this PR, but that reminds me we should also add the list view types here (maybe you can do that in one of the open/future PRs you are still doing for list view) |
After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 3f7b288. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them. |
This was referenced Aug 11, 2026
AlenkaF pushed a commit
that referenced
this pull request
Aug 27, 2026
) ### Rationale for this change `pyarrow.types.is_nested()` said a run-end encoded type wasn't nested, while the C++ `arrow::is_nested()` says it is: ```python >>> import pyarrow as pa >>> t = pa.run_end_encoded(pa.int32(), pa.string()) >>> t.num_fields 2 >>> pa.types.is_nested(t) False ``` The Python side reads from a hardcoded `_NESTED_TYPES` set in `python/pyarrow/types.py` rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against `is_nested()` in `cpp/src/arrow/type_traits.h`, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there. It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers `True` here. This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in. ### What changes are included in this PR? Adds `Type_RUN_END_ENCODED` to `_NESTED_TYPES`, which is the whole fix. In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in. ### Are these changes tested? Yes. `test_is_nested_or_struct` fails on the current code and passes with the change. I don't have a local C++ build, so I checked this by running the updated `test_types.py` against an installed pyarrow 25.0.0 with the same one-line change applied to its `types.py`. Before the change that file had 87 passing with `test_is_nested_or_struct` failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the `pickle_module` fixture, which comes from a conftest I wasn't loading, and the failure is `test_pytz_timezone_roundtrip`. ### Are there any user-facing changes? Yes, though it's small. `pa.types.is_nested()` now returns `True` for run-end encoded types where it previously returned `False`. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads `is_nested` or `_NESTED_TYPES`, so the effect is limited to callers. * GitHub Issue: #50847 Authored-by: nishad shabbir <nishadshabbir28@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
What changes are included in this PR?
This PR fixes a minor bug in
types.is_nestedwhich doesn't consider theFIXED_SIZE_LISTtype as nested type.Are these changes tested?
Are there any user-facing changes?