[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.

[davlee1972]

Describe the bug, including details regarding any error messages, version, and platform.

I'm getting the following error trying to write a parquet file.

pyarrow.lib.ArrowInvalid: Column '_Id' is declare non-nullable, but contains nulls.

I think this is happening because there might be actual nulls in the column for the purpose of making a struct array even between the outer struct and inner struct. The image below is showing is_valid: [false] which looks odd to begin with..

The entire ShareClassType field is nullable and it contains a NULL, but the writer is complaining that a non-nullable field in belonging to non-nullable inner struct contains NULLs.

Here is the schema for this item:

and the data in it:

Component(s)

C++, Python

[davlee1972]

As a temporary work around.. I was able to write the pyarrow table using Datafusion..

Read it back using pyarrow and the rewrite it back as a rowgroup..

Datafusion changed the underlying data to [""].

Schema:

Data:

[davlee1972]

In PyArrow, a NULL column inside a StructArray is often initialized with a default value like "" (empty string) or 0. This happens because Arrow does not allocate null buffers for nested struct fields; instead, it puts a placeholder in the null slot.To fix this, use PyArrow compute functions to replace the empty strings with proper null values.

I guess using pa.null() or [None] to even out nested struct arrays won't work? I should be adding empty strings instead even though an empty string technically isn't a struct data type?

This could get complex since the outer struct has both scalar types and struct types..

I think this should be fixed in the parquet writer.. The [null] value appear to be perfectly valid in the pyarrow table..

[fenfeng9]

A null parent struct masks its child slots. Therefore, an underlying Arrow child-array slot may be null when the parent struct is null, even if the child field is non-nullable. This is valid because the child is logically absent. The Parquet writer incorrectly rejects this case.

Reproduce

import pyarrow as pa
import pyarrow.parquet as pq


value_field = pa.field("value", pa.int32(), nullable=False)

# The physical child slot is null.
value_column = pa.array([None], type=pa.int32())

# True means that the corresponding parent struct is null.
parent_is_null = pa.array([True])

array = pa.StructArray.from_arrays(
    arrays=[value_column],
    fields=[value_field],
    mask=parent_is_null,
)

print("logical array:", array.to_pylist())
print("physical child:", array.field("value").to_pylist())

pq.write_table(pa.table({"data": array}), "output.parquet")

Result

logical array: [None]
physical child: [None]
Traceback (most recent call last):
  File "/path/to/test.py", line 22, in <module>
    pq.write_table(pa.table({"data": array}), "output.parquet")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/site-packages/pyarrow/parquet/core.py", line 2015, in write_table
    writer.write_table(table, row_group_size=row_group_size)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/site-packages/pyarrow/parquet/core.py", line 1180, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_parquet.pyx", line 2397, in pyarrow._parquet.ParquetWriter.write_table
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 'value' is declared non-nullable but contains nulls

Analysis

This appears to be a regression introduced by #44921.

The check uses leaf_array.null_count() without accounting for null parent structs, so it incorrectly rejects child nulls masked by a null parent.

arrow/cpp/src/parquet/column_writer.cc

Lines 1432 to 1442 in 5815dc0

	if (!leaf_field_nullable && leaf_array.null_count() != 0) {
	return Status::Invalid("Column '", descr_->name(),
	"' is declared non-nullable but contains nulls");
	}
	bool maybe_parent_nulls = level_info_.HasNullableValues() && !single_nullable_element;
	if (maybe_parent_nulls) {
	ARROW_ASSIGN_OR_RAISE(
	bits_buffer_,
	::arrow::AllocateResizableBuffer(
	bit_util::BytesForBits(properties_->write_batch_size()), ctx->memory_pool));
	bits_buffer_->ZeroPadding();

[fenfeng9]

take