GH-39914: [pyarrow] Fall back to schema field type for unparseable complex dtype metadata

[tritsystem]

Rationale for this change

pd.read_parquet() (and Table.to_pandas() generally, with no
types_mapper/dtype_backend="pyarrow") raises TypeError for any
column whose pandas metadata records a complex ArrowDtype -- list,
struct, or dictionary -- as its numpy_type, e.g.
"list<item: string>[pyarrow]". This has been broken since the metadata
round-tripping was introduced, and is the long-standing root cause of
pandas-dev/pandas#53011 (open since 2023, still reproducing today with
no workaround at the pandas level).

importpandasaspdimportpyarrowaspadata=pd.DataFrame({
"a": pd.Series([["x"], ["x", "y"]], dtype=pd.ArrowDtype(pa.list_(pa.string()))),
})
data.to_parquet("data.parquet") # SUCCESSpd.read_parquet("data.parquet") # TypeError: data type 'list<item: string>[pyarrow]' not understood

#39914 (this PR's linked issue) previously fixed the case where an
explicit types_mapper / dtype_backend="pyarrow" is passed at read
time -- #44720 resolved those columns earlier in _get_extension_dtypes,
so the buggy string-parsing branch is skipped entirely for that path.
The default backend, and any metadata-only path with no
types_mapper, still hit the original crash -- which is why the pandas
issue has stayed open and users keep re-discovering it (see the thread's
comments through 2025-08).

What changes are included in this PR?

In _get_extension_dtypes(), when pandas_dtype(dtype) raises
TypeError on the metadata's numpy_type string, fall back to building
the ArrowDtype directly from the schema's actual field type
(table.schema.field(name).type) instead of propagating the error. The
schema always carries the precise type regardless of whether the
metadata string happens to be parseable, and this is exactly the pattern
several commenters on the pandas issue independently proposed (e.g.
this comment)
but that was never implemented in a merged fix.

Are these changes tested?

Added test_to_pandas_extension_dtypes_mapping_complex_type_no_types_mapper,
covering both a list and a struct ArrowDtype column round-tripped
through Table.to_pandas() with notypes_mapper -- the exact case
that previously crashed. Confirmed red on unpatched code / green after
the fix.

Also manually verified (not just the new test) against pandas 3.0.4 +
pyarrow 24.0.0:

• list, struct, and dictionary ArrowDtype columns round-trip through
  both the default backend and dtype_backend="pyarrow".
• A double write/read/write/read round-trip of a struct column (the
  scenario from this comment)
  still works.
• Plain numpy dtypes and simple extension dtypes (Int64) are
  unaffected -- no regression to the already-working paths.

Are there any user-facing changes?

This PR contains a "Critical Fix".
This makes pd.read_parquet(...) (default backend, no
dtype_backend="pyarrow" needed) work for any DataFrame containing a
complex ArrowDtype column (list/struct/dictionary), which currently
raises an opaque TypeError and has no workaround short of stripping the
pandas metadata before writing.

• GitHub Issue: [Python][Parquet] Parquet files created from Pandas dataframes with Arrow-backed list columns cannot be read by pd.read_parquet #39914

AI Generation Disclosure

Generated with Claude Code. The bug
was found by searching for confirmed, still-open, high-severity issues
across large open-source projects; root-caused and fixed by tracing the
actual pandas_compat.py source against the linked pandas issue and the
prior partial fix (#44720), then verified against a fresh install of the
latest pandas/pyarrow rather than assumed from the issue thread alone.

🤖 Generated with Claude Code

[github-actions]

⚠️ GitHub issue #39914has been automatically assigned in GitHub to PR creator.