GH-38809: [Python] Add tests for struct type promotion in concat_tables by slachiewicz · Pull Request #50956 · apache/arrow

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

slachiewicz · 2026-08-23T10:07:18Z

Rationale for this change

GH-38809 reports that pa.concat_tables cannot promote struct types. That is no longer true: the snippet in the issue works from 19.0.0 onwards. Schema unification of structs was there from the start (MergeStructs in cpp/src/arrow/type.cc, added in #36846); what was missing was the cast side, which GH-44555 / #44587 fixed by letting CastStruct fill absent output fields with nulls. #45246 later added field reordering.

None of that is covered by a Python test — the existing test_concat_tables_with_promotion* tests use flat types only — so the behaviour could regress unnoticed. These tests pin it down and give GH-38809 something to close on.

What changes are included in this PR?

Three tests in python/pyarrow/tests/test_table.py:

test_concat_tables_with_struct_promotion — the exact case from [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809: a struct field present in one table only is filled with null, null-typed fields are promoted, and the unified struct keeps the first table's field order.
test_concat_tables_with_nested_struct_promotion — int32 and float64 nested inside a struct promote to float64 under permissive, and still fail under default.
test_concat_tables_with_list_of_struct_promotion — the same promotion below a list.

No library changes.

Are these changes tested?

They are tests.

Verified: pytest -k struct_promotion against pyarrow 25.0.1 → 3 passed. The first case fails on 18.1.0 with exactly the error quoted in GH-38809 (struct fields don't match or are in the wrong order) and passes on 19.0.0, which is how the fixing release was identified.

Are there any user-facing changes?

No.

This change was created with AI assistance. The version bisect, the assertions and the claim about which commit fixed it were each checked by running them; the expected values in the tests are the observed output of concat_tables, not hand-written guesses.

GitHub Issue: [Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809

…t_tables Struct promotion works since 19.0.0 (apacheGH-44555 taught the struct cast to fill absent output fields with null), but no Python test covered it. Verified: pytest -k struct_promotion against pyarrow 25.0.1 -> 3 passed; the first case fails on 18.1.0 with the error from apacheGH-38809.

github-actions · 2026-08-23T10:07:41Z

⚠️ GitHub issue #38809has been automatically assigned in GitHub to PR creator.

uros-b · 2026-08-24T08:25:01Z

Thank you @slachiewicz!

github-actionsBot added Component: Python awaiting review Awaiting review labels Aug 23, 2026

slachiewicz mentioned this pull request Aug 23, 2026
[Python] pyarrow.concat_tables did not really enable the unify_schemas, it only supply unify_schemas options but not actually enable it #38809
Open

slachiewicz marked this pull request as ready for review August 23, 2026 10:13

slachiewicz requested review from AlenkaF, raulcd and rok as code owners August 23, 2026 10:13

uros-b approved these changes Aug 24, 2026
View reviewed changes

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 24, 2026