Uh oh!
There was an error while loading. Please reload this page.
GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist - #50327
Conversation
…arrays Array.to_pylist() converts one element at a time: each row allocates a C++ Scalar (Array::GetScalar), a Python Scalar wrapper and, for list types, a Python Array wrapper for the row's values slice plus a fresh generator, before recursing per element. On top of the allocation cost itself, these GC-tracked wrappers repeatedly trigger collections that traverse the growing result list (~20% of runtime). This makes to_pylist on list-typed arrays several times slower than the bulk to_pandas conversion path. Add bulk to_pylist overrides: * ListArray / LargeListArray / FixedSizeListArray convert the referenced range of child values with a single recursive to_pylist call, then slice the resulting Python list per row using the raw offsets and the validity bitmap. MapArray keeps the generic scalar-based path (association-tuple / maps_as_pydicts duplicate-key semantics), as do the list-view types (overlapping views must not share sublist objects). * StringArray / LargeStringArray decode values directly from the data buffer (GetValue + PyUnicode_DecodeUTF8), matching StringScalar.as_py (= str(buf, 'utf8')) exactly. Semantics are unchanged; values inside numeric lists stay Python ints/None. Benchmarks (M4 Max, 2M rows of 2-element lists / 1M rows nested): list<string> 1.93s -> 0.34s, list<list<int32>> 2.10s -> 0.65s, flat string (4M) 0.83s -> 0.05s. Co-authored-by: Isaac
There was a problem hiding this comment.
Pull request overview
This PR improves pyarrow.Array.to_pylist() performance for list-like arrays and (large) string arrays by adding specialized bulk conversion implementations that avoid per-element Scalar allocation and wrapper overhead, while keeping output semantics unchanged.
Changes:
- Add bulk
to_pylist()implementations forListArray,LargeListArray, andFixedSizeListArraythat convert child values once and slice per row using offsets. - Add fast
to_pylist()implementations forStringArrayandLargeStringArraythat decode directly from the value buffer. - Add a new test validating bulk-path results against the scalar-based reference across nested, sliced, empty, and all-null inputs.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/pyarrow/array.pxi | Adds type-specific to_pylist() fast paths for list-like and string arrays. |
| python/pyarrow/tests/test_array.py | Adds a regression/differential test to ensure bulk paths match scalar-based conversion. |
| n = arr.length() | ||
| result = [] | ||
| # Decode values straight from the data buffer instead of creating | ||
| # a C++ Scalar and a Python Scalar wrapper per value (see GH-28694). | ||
| if arr.null_count() == 0: | ||
| for i in range(n): | ||
| data = arr.GetValue(i, &length) | ||
| result.append( | ||
| cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)) | ||
| else: | ||
| for i in range(n): | ||
| if arr.IsNull(i): | ||
| result.append(None) | ||
| else: | ||
| data = arr.GetValue(i, &length) | ||
| result.append( | ||
| cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)) | ||
| return result |
There was a problem hiding this comment.
null_count() is a one-time vectorized popcount over the validity bitmap (~n/8 bytes, well under a millisecond for 2M rows), computed and cached per ArrayData. In exchange, the no-null branch skips the per-element IsNull() check entirely. Branching on null_bitmap_data() == NULL instead would save that single scan but degrade the common case of a sliced/combined array that has a bitmap yet contains no nulls in range — that would take the per-element IsNull() path forever. So the current form should be the better trade-off in practice.
gaogaotiantian
commented
Jul 1, 2026
I'm not an expert in Cython but curious about how It would be an interesting experiment to do a full-allocation for the list before assigning the data, as we already know the length of the list. An extra step forward is to declare the return value as a list in Cython so it can optimize Something like cdef list result = [None] * n
cdef Py_ssize_t i
for i inrange(n):
result[i] = ...
return resultFor a long list this might push the performance even further. |
viirya
commented
Jul 2, 2026
Good idea — I tried exactly that ( Two reasons, I think: Cython already lowers |
gaogaotiantian
commented
Jul 2, 2026
Okay if the benchmark is similar this is good. Would defining result as a list help? Like a cdef list for it and do append. Just curious whether cython knows it's a list already - maybe it does and that's why it's fast. |
viirya
commented
Jul 2, 2026
Good question — Cython already knows. Its type inference marks |
pitrou
commented
Jul 8, 2026
I agree this is extremely wasteful, but the approach here seems very ad hoc and also solves the performance issue for a very limited set of types, while the performance problem is more general: In [5]: a=np.arange(10_0000)
In [6]: %timeita.tolist()
722μs ± 43.8μsperloop (mean ± std. dev. of7runs, 1,000loopseach)
In [7]: b=pa.array(a)
In [8]: %timeitb.to_pylist()
17.5ms ± 147μsperloop (mean ± std. dev. of7runs, 100loopseach)
In [9]: b.to_pylist() ==a.tolist()
Out[9]: TrueHow about we do something like the following:
|
viirya
commented
Jul 8, 2026
Thanks for the review, @pitrou! Agreed — the per-type
Types with non-trivial One question before I rework: do you prefer this mechanism in Cython as sketched, or as a C++ |
pitrou
commented
Jul 8, 2026
Doing this in Cython would probably be more maintainable, right? Wrangling CPython API calls from C++ is generally tedious and error-prone. |
…to_pylist Per review feedback, replace the per-type to_pylist overrides with a general mechanism: * Array gains a cdef _getitem_py(i) returning self[i] as a Python object. The base implementation is GetScalar + Scalar.as_py, so any type without a specialization behaves exactly as before. * The baseline Array.to_pylist becomes a single loop over _getitem_py. maps_as_pydicts != None keeps the Scalar-based path (map->dict conversion has per-entry duplicate-key semantics). * Specializations avoid all per-element Scalar and per-row Array wrapper allocation: integers/floats (type_id switch on NumericArray; dates/times/timestamps fall through to the exact base), boolean, string/binary (+ large variants), list/large_list/fixed_size_list (per-row list built from the child's _getitem_py over the offset range, child wrapper cached on the array), map (list of key/value tuples), struct (dict per row; duplicate field names fall back so they raise ValueError like StructScalar.as_py). Benchmarks (M4 Max): flat int64 with nulls 4M ~0.39s -> 0.028s (~7ns per element, on par with ndarray.tolist); flat string 4M 0.83s -> 0.06s; list<string> 2M 1.93s -> 0.46s; list<list<int32>> 1M 2.10s -> 0.40s; struct<int64,string> 1M 0.91s -> 0.07s; map<string,int64> 1M 2.77s -> 0.74s. Co-authored-by: Isaac
viirya
commented
Jul 8, 2026
Reworked as suggested — the PR now adds a scalar-free Your Net diff is smaller than the previous approach (−104 lines) since the per-type |
Co-authored-by: Isaac
| cdef: | ||
| shared_ptr[CArray] sp_array | ||
| CArray* ap | ||
| # Lazily wrapped child array(s) reused by _getitem_py (see GH-50326) | ||
| object _children_cache |
There was a problem hiding this comment.
Good point — moved the declaration after the pre-existing attributes in 728584f, so type/_name offsets stay stable and the new field follows the append-only convention that Cython's size check assumes.
Uh oh!
There was an error while loading. Please reload this page.
…ssertion These test changes were described in 3d303ce's commit message but test_array.py was accidentally left out of that commit. Also add a TODO noting that maps_as_pydicts currently falls back to the Scalar path for the whole array even when no maps are involved (addressed by apacheGH-50429). Co-authored-by: Isaac
pitrou
commented
Jul 22, 2026
@github-actions crossbow submit -g python |
Revision: 1d80beb Submitted crossbow builds: ursacomputing/crossbow @ actions-08c8d9798b |
…meter as int64_t The C++ signatures take int64_t; the previous 'int' declarations made Cython silently truncate 64-bit row indices before the call, which the new _getitem_py list/map paths can hit for arrays longer than INT_MAX. Align all list-like/binary/union declarations with the C++ headers. Co-authored-by: Isaac
viirya
commented
Jul 22, 2026
@pitrou Heads-up: I pushed one more small commit (9b1863d) after your approval — the Copilot review on the stacked #50430 noticed that the |
pitrou
commented
Jul 22, 2026
Thank you! I don't think it will change anything for CI, so I'll merge accordingly. |
…rations with int64_t Follow-up to 9b1863d, which only fixed value_offset/value_length: sweep the rest of the per-element accessors used (directly or not) with 64-bit indices — CArray.IsNull, the numeric/boolean/temporal Value(), binary/fixed-size-binary GetValue(), GetString() and decimal FormatValue() — all of which take int64_t in the C++ headers. In particular IsNull is called with an int64_t index in every new _getitem_py implementation. Co-authored-by: Isaac
viirya
commented
Jul 22, 2026
@pitrou One more declaration-alignment commit (95f29b9), prompted by further Copilot findings on the stacked #50430: my earlier fix only covered |
viirya
commented
Jul 22, 2026
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 0083d11. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them. |
Rationale for this change
pa.Array.to_pylist()converts one element at a time throughArray::GetScalarplus a PythonScalarwrapper; for list types each row additionally allocates a PythonArraywrapper for the row's values slice and a fresh generator before recursing per element. Asampleprofile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% inGetScalar, and only ~7% doing the useful work of creating the output objects — makingto_pylistseveral times slower than converting viato_pandas()and back, and ~24x slower thanndarray.tolist()for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943).What changes are included in this PR?
Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type
to_pylistoverrides:Arraygainscdef object _getitem_py(self, int64_t i), returningself[i]as a Python object. The base implementation isGetScalar+Scalar.as_py, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...).Array.to_pylistbecomes a single loop over_getitem_py.maps_as_pydicts != Nonekeeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics.type_idswitch onNumericArray; date/time/timestamp subclasses fall through to the exact base),GetValue+PyUnicode_DecodeUTF8/PyBytes_FromStringAndSize, matchingstr(buf, 'utf8')/to_pybytes()exactly),_getitem_pyover the offset range; the wrapped child is cached on the parent array),MapScalar.as_py),ValueErrorlikeStructScalar.as_py).Nested types compose without any per-row wrappers.
ChunkedArray.to_pylist,Table.to_pylistandListScalar.as_pydelegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32.Benchmarks (macOS arm64, M4 Max):
int64with nulls (4M)ndarray.tolist)string(4M)list<string>(2M rows)list<list<int32>>(1M rows)struct<int64,string>(1M rows)map<string,int64>(1M rows)Are these changes tested?
test_to_pylist_bulk_paths(added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against[x.as_py() for x in arr]with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-nameValueError), map (incl. strict-mode duplicate-keyKeyError), dictionary/null fallbacks, sliced/chunked arrays, and bothmaps_as_pydictsmodes — no differences.pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py: 1295 passed.Are there any user-facing changes?
No behavior changes, only performance.
This pull request and its description were written by Isaac.