Fix null list of structs being read and written as an empty list

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.

[slachiewicz]

A null list<struct<...>> was rebuilt as an empty list, losing the distinction
between the two. test_null_list_and_map has documented this since #252, asserting
the corrupted value with the correct assertion commented out pending
apache/arrow#38809.

That dependency turns out not to hold for this symptom: from_arrays takes a mask
argument, so carrying the validity bitmap over fixes the null loss on its own. The
rebuild itself is still needed for #38809 and is unchanged.

The write path is affected too, and there the loss is permanent — the Parquet file
pyiceberg produces contains an empty list, so no reader can tell it from a null. The
new unit test covers that direction; the integration assertion covers the read side.

Closes#3833, which has the full analysis and a standalone reproduction.

This change was created with AI assistance.

[slachiewicz]

Pushed a second commit that removes the # This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809 comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for.

The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while cast matches struct fields by name. Dropping the rebuild and letting the trailing list_array.cast(arrow_field) do the work silently nulls out any field renamed inside a list element:

with rebuild: [[{'after': 1}], [], None, [{'after': 3}]]
rebuild removed: [[{'after': None}], [], None, [{'after': None}]]

That is test__to_requested_schema_renamed_field_in_list_of_structs, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own.

This comment was created with AI assistance.