GH-50312: [Python] Fix UUID extension type round-trip to pandas returning bytes

[parker-cassar]

Rationale for this change

Converting a Table with an arrow.uuid extension column to pandas currently produces a column of bytes instead of uuid.UUID objects. Without a dedicated conversion path, Table.to_pandas() / Array.to_pandas() falls back to the storage type (fixed_size_binary(16)) and materializes raw bytes. Downstream callers (and tests such as those landing via pandas UUID Parquet coverage) expect uuid.UUID objects, matching what to_pylist() / UuidScalar.as_py() already return.

Note: the original issue suggested this might be specific to Python 3.14, but the same bytes result reproduces on Python 3.10-3.14 with current PyArrow - UuidType never had a pandas conversion path.

Closes#50312

What changes are included in this PR?

Convert UUID extension arrays to uuid.UUID objects in the C++ pandas conversion path (integrated from @rok's proposal in rok#55), rather than relying on a Python to_pandas_dtype() / __from_arrow__ compat layer:

• Construct uuid.UUID objects from storage bytes in helpers.cc (UuidFromBytes) and use that from arrow_to_pandas.cc
• Reuse empty kwargs / shared empty tuple constants so conversion does not allocate per element
• Keep to_numpy(zero_copy_only=False) on UUID extension arrays returning storage bytes (extension to NumPy should follow storage type)
• Add/extend regression tests for pandas round-trip and NumPy behavior

Are these changes tested?

Yes.

Acceptance criteria:

• Tests added for the UUID to pandas round-trip (and NumPy storage behavior)
• Relevant local / CI Python tests passing for the touched paths
• Follows project style / review feedback (lint fix included)
• No intentional breaking public API change beyond correcting incorrect bytes return values to uuid.UUID for to_pandas()

Before / after evidence (backend; console):

# Before (repro on Python 3.10-3.14 with unfixed PyArrow)
>>> type(result_df.loc[0, "id"])
<class 'bytes'>
# After (with this PR)
>>> type(result_df.loc[0, "id"])
<class 'uuid.UUID'>

Also covered by added unit tests in python/pyarrow/tests/parquet/test_data_types.py and python/pyarrow/tests/test_extension_type.py.

Are there any user-facing changes?

Yes. Table.to_pandas() / Array.to_pandas() now returns uuid.UUID for arrow.uuid columns instead of bytes. to_numpy(zero_copy_only=False) continues to expose storage bytes.

• GitHub Issue: [Python][Parquet] FIXED_LEN_BYTE_ARRAY fails to cast to UUID on Python 3.14 / Nightly builds #50312

[github-actions]

⚠️ GitHub issue #50312has been automatically assigned in GitHub to PR creator.

[GiTaDi-CrEaTe]

Hey @parker-cassar, thanks so much for jumping on this so quickly. Just out of curiosity—since this delegates to to_pylist(), do you think this pattern might end up being useful for other Arrow extension types down the way???

[rok]

Thanks for working on this @parker-cassar!
A couple of questions since we're aiming to support Pandas 3.x and Pandas 2.x too for a while.

[parker-cassar]

Hey @parker-cassar, thanks so much for jumping on this so quickly. Just out of curiosity—since this delegates to to_pylist(), do you think this pattern might end up being useful for other Arrow extension types down the way???

Maybe! It works here because the target is object dtype anyway, so building the Python objects is unavoidable. Other extension types could use the same from_arrow + reshape structure, but anything performance-sensitive would want a zero-copy path instead of going through Python objects.

[parker-cassar]

@rok Applied both of your suggestions. Ready for another look whenever you have time.

[rok]

@parker-cassar thank you for your patience :)

I've been thinking about this a bit and perhaps we could convert to UUID in c++ and avoid awkward Python compat layer? Pandas had a UUID type proposal that seems stalled but could get completed. See my proposal what this could look like. Let me ask for some feedback from others before you continue.
@pitrou@jorisvandenbossche@raulcd@AlenkaF

[rok]

@parker-cassar I think we better implement this in C++ kind of like proposed here. This will likely improve performance and give us a cleaner API/make compat layer with pandas neater. Sorry for not suggesting this sooner. Do you think you can continue in this vein?

[parker-cassar]

@parker-cassar I think we better implement this in C++ kind of like proposed here. This will likely improve performance and give us a cleaner API/make compat layer with pandas neater. Sorry for not suggesting this sooner. Do you think you can continue in this vein?

Yes I agree and am happy you suggested this (: the Python compat layer approach doesn't really make sense here, especially since a UUID column in pandas isn't exactly a rare case. Doing it natively is cleaner. I'll give it a go and I'll try to have a PR in a few days or less based on your proposal.

[parker-cassar]

@rok Quick update: your proposal is integrated and the full round-trip passes. I'm still working on my last pass and I will have it up in the next day or two. Thanks for the proposal, made it a lot easier.

[rok]

I have a potential performance improvement suggestion, but other than that this looks pretty good.

[rok]

I think we're almost there - just the matter of ensuring type(uuid_array.to_numpy(zero_copy_only=False)[0]) is bytes, since extension-array NumPy conversion should delegate to its storage type.

[parker-cassar]

@rok The macOS 15-intel check failed in the Install MinIO step (wget couldn't resolve dl.min.io) before anything was built or tested. Are you able to re-run that job?

[rok]

LGTM. I'll wait a bit if @AlenkaF or @pitrou have time to review.

[AlenkaF]

Thank you for working on this @parker-cassar and providing the C++ version of the solution. I think the PR looks good and am happy to see it get merged!

[parker-cassar]

Thanks @AlenkaF! Big thanks to @rok too!

[parker-cassar]

@rok@AlenkaF just wondering if there's anything left on my end before this can be merged? Thanks!

[rok]

Sorry for the wait @parker-cassar and thank you for your flexibility and patience!
I think this is ready and will merge.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 892c73d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.