Describe the bug, including details regarding any error messages, version, and platform.
Casting a map array still hard-aborts the process (SIGABRT, not a raised exception) with Map array keys array should have no nulls when the keys child carries a validity bitmap — even an all-valid one. This is a residual case of #38553: the fix in #41957 changed MapArray::ValidateChildData to use MayHaveNulls(), which resolved the no-bitmap repro on that issue, but MayHaveNulls() answers true whenever a validity bitmap is present and the null count is unknown — it never counts. A cast's output starts with an unknown null count, so the fatal ARROW_CHECK_OK(ValidateChildData(...)) in MapArray::SetData (array_nested.cc:912 as of current main) kills the process.
Three-line reproduction, all public API, on pyarrow 24.0.0 (the current release):
importpyarrowaspam=pa.array([{"a": "1"}, {"b": "2"}], type=pa.map_(pa.string(), pa.string()))
m.filter(pa.array([False, True])).cast(pa.map_(pa.large_string(), pa.large_string()))/Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/array_nested.cc:912: Check failed: _s.ok() Operation failed: ValidateChildData(data->child_data)
Bad status: Invalid: Map array keys array should have no nulls
Exit code 134. The filter call itself succeeds and the filtered array passes validate(full=True).
filter matters only because it materializes new child arrays that carry an all-valid validity bitmap; slice re-views the original buffers without one, which is why the slice→cast variant works after #41957. The bitmap alone is sufficient — same abort with no filter involved:
importpyarrowaspakeys=pa.array(["a", "b"])
validity=pa.array([True, True]).buffers()[1] # all-valid bitmapkeys=pa.Array.from_buffers(pa.string(), 2, [validity, keys.buffers()[1], keys.buffers()[2]])
m=pa.MapArray.from_arrays(pa.array([0, 1, 2], type=pa.int32()), keys, pa.array(["1", "2"]))
m.cast(pa.map_(pa.large_string(), pa.large_string())) # aborts
Forcing null-count computation on the input (m.keys.null_count → 0) does not help: the cast produces a fresh keys ArrayData whose count is unknown again.
Real-world impact: pyiceberg's scan path does filter-then-cast (row-filtered Parquet read, then cast to large types), so any row-filtered scan that projects a map column kills the worker process.
Two candidate fixes, possibly both:
MapArray::ValidateChildData (array_nested.cc:899/905) could force-compute the null count (GetNullCount() != 0) rather than ask MayHaveNulls(); counting an all-valid bitmap is cheap next to aborting the process.- The
ARROW_CHECK_OK at array_nested.cc:912 turns an Invalid status into process death even when the data is genuinely invalid; a returned/raised error would be recoverable.
Environment: pyarrow 24.0.0 (PyPI wheel), Python 3.13.13, macOS 26.4.1 arm64.
Component(s)
C++, Python
🤖 — posted via Claude Code
Describe the bug, including details regarding any error messages, version, and platform.
Casting a map array still hard-aborts the process (SIGABRT, not a raised exception) with
Map array keys array should have no nullswhen the keys child carries a validity bitmap — even an all-valid one. This is a residual case of #38553: the fix in #41957 changedMapArray::ValidateChildDatato useMayHaveNulls(), which resolved the no-bitmap repro on that issue, butMayHaveNulls()answers true whenever a validity bitmap is present and the null count is unknown — it never counts. A cast's output starts with an unknown null count, so the fatalARROW_CHECK_OK(ValidateChildData(...))inMapArray::SetData(array_nested.cc:912as of current main) kills the process.Three-line reproduction, all public API, on pyarrow 24.0.0 (the current release):
Exit code 134. The
filtercall itself succeeds and the filtered array passesvalidate(full=True).filtermatters only because it materializes new child arrays that carry an all-valid validity bitmap;slicere-views the original buffers without one, which is why the slice→cast variant works after #41957. The bitmap alone is sufficient — same abort with no filter involved:Forcing null-count computation on the input (
m.keys.null_count→ 0) does not help: the cast produces a fresh keysArrayDatawhose count is unknown again.Real-world impact: pyiceberg's scan path does filter-then-cast (row-filtered Parquet read, then cast to large types), so any row-filtered scan that projects a map column kills the worker process.
Two candidate fixes, possibly both:
MapArray::ValidateChildData(array_nested.cc:899/905) could force-compute the null count (GetNullCount() != 0) rather than askMayHaveNulls(); counting an all-valid bitmap is cheap next to aborting the process.ARROW_CHECK_OKatarray_nested.cc:912turns anInvalidstatus into process death even when the data is genuinely invalid; a returned/raised error would be recoverable.Environment: pyarrow 24.0.0 (PyPI wheel), Python 3.13.13, macOS 26.4.1 arm64.
Component(s)
C++, Python
🤖 — posted via Claude Code