GH-39914: [pyarrow] Reorder to_pandas extension dtype mapping

[bretttully]

Rationale for this change

This is a long standing pandas ticket with some fairly horrible workarounds, where complex arrow types do not serialise well to pandas as the pandas metadata string is not parseable. However, types_mapper always had highest priority as it overrode what was set before.

What changes are included in this PR?

By switching the logical ordering, it means that we don't need to call _pandas_api.pandas_dtype(dtype) when using the pyarrow backend, thus resolving the issue of complex dtype with list or struct. It will likely still fail if the numpy backend is used, but at least this gives a working solution rather than an inability to load files at all.

Are these changes tested?

Existing tests should stay unchanged and a new test for the complex type has been added

Are there any user-facing changes?

This PR contains a "Critical Fix".
This makes pd.read_parquet(..., dtype_backend="pyarrow") work with complex data types where the metadata added by pyarrow during pd.to_parquet is not serialisable and currently throwing an exception. This issue currently prevents the use of pyarrow as the default backend for pandas.

• GitHub Issue: [Python][Parquet] Parquet files created from Pandas dataframes with Arrow-backed list columns cannot be read by pd.read_parquet #39914

[github-actions]

❌ GitHub issue #53011 could not be retrieved.

[github-actions]

⚠️ GitHub issue #39914has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

By switching the logical ordering, it means that we don't need to call _pandas_api.pandas_dtype(dtype) when using the pyarrow backend,

And because you added a name not in ext_columns to the subsequent methods to fill ext_columns, this should preserve the priority of the different methods to determine the pandas dtype? (metadata < pyarrow extension type < types_mapper)

[bretttully]

By switching the logical ordering, it means that we don't need to call _pandas_api.pandas_dtype(dtype) when using the pyarrow backend,

And because you added a name not in ext_columns to the subsequent methods to fill ext_columns, this should preserve the priority of the different methods to determine the pandas dtype? (metadata < pyarrow extension type < types_mapper)

Yes, exactly. Priority remains the same, but functions are skipped if the field already has a type, meaning that the code causing the error is no longer called if types_mapper is provided.

[jorisvandenbossche]

The test_dlpack failure in the tests you can ignore (#44728)

[jorisvandenbossche]

Looks good!
I triggered CI again

[bretttully]

Thanks @jorisvandenbossche -- is the process that I can merge this following approval, or is that done by a core maintainer?

[raulcd]

is the process that I can merge this following approval, or is that done by a core maintainer?

A committer will merge, probably @jorisvandenbossche in this specific case, once everything is running and addressed. I've triggered CI for the latest changes.

[jorisvandenbossche]

@github-actions crossbow submit -g python

[github-actions]

Revision: e3b9892

Submitted crossbow builds: ursacomputing/crossbow @ actions-e01b93275b

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[jorisvandenbossche]

@raulcd it seems something is going wrong with the minimal test builds (eg example-python-minimal-build-fedora-conda). The logs indicate "Successfully installed pyarrow-0.1.dev16896+ge3b9892", which then messes up pandas detection of the pyarrow version (for the pyarrow integration in pandas, pandas checks if pyarrow is recent enough and otherwise errors), giving some test failures.

(but also not entirely sure how this PR causes this issue, since I don't see the nightlies fail for the minimal builds at the moment)

[jorisvandenbossche]

(the other failures are the known nightly dlpack failures)

[raulcd]

The logs indicate "Successfully installed pyarrow-0.1.dev16896+ge3b9892"

From the git checkout I see is pulling from the remote on Syncing repository: bretttully/arrow. I recall an issue if dev tags are not present we are unable to detect the correct version. The remote doesn't seem to have other branches and/or tags.

[raulcd]

I've opened an issue because we should find a way to not fail if the dev tag is not present:

• [Python] Jobs fail if Pyarrow version is not correctly generated due to missing remote dev tags #44803

[jorisvandenbossche]

Thanks for investigating that!

So then to resolve this here, @bretttully should fetch the upstream tags and push that to his fork? Something like

git fetch upstream
git push origin --tags

(assuming upstream is apache/arrow and origin is bretttully/arrow)

[bretttully]

I have merged upstream/main and pushed tags. Let's see if this works...

[raulcd]

@github-actions crossbow submit example-python-minimal-build-*

[github-actions]

Revision: 685167f

Submitted crossbow builds: ursacomputing/crossbow @ actions-524e782c26

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv

[bretttully]

Is there anything else for me to do here?

[raulcd]

Is there anything else for me to do here?

I don't think so. I am not comfortable with this area of our codebase so I'll let @jorisvandenbossche merge once he's happy about it, but as he already approved, he might do that soon.

[jorisvandenbossche]

Is there anything else for me to do here?

No, just me getting back to merge it!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 8548c22.

There were 132 benchmark results with an error:

• Commit Run on arm64-t4g-2xlarge-linux at 2024-11-27 18:04:38Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-07, scale_factor=1
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-09, scale_factor=1
• and 130 more (see the report linked below)

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

[bretttully]

🚀 Thanks for all your help here @jorisvandenbossche!