GH-39010: [Python] Introduce maps_as_pydicts parameter for to_pylist, to_pydict, as_py

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.

[jonasdedden]

Rationale for this change

Currently, unfortunately MapScalar/Array types are not deserialized into proper Python dicts, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds.

What changes are included in this PR?

A new parameter maps_as_pydicts is introduced to to_pylist, to_pydict, as_py which will allow proper roundtrips:

import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]

Are these changes tested?

Yes. There are tests for to_pylist and to_pydict included for pyarrow.Table, whilst low-level MapScalar and especially a nesting with ListScalar and StructScalar is tested.

Also, duplicate keys now should throw an error, which is also tested for.

Are there any user-facing changes?

Yes. The as_py() method on Scalar instances can be called with a new keyword argument maps_as_pydicts.

As a consequence, if you implement your own Scalar subclass (for example for an extension type), you should change its signature to accept that new argument. For example this definition:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returndeserialize_json(self.value.as_py() ifself.valueelseNone)

could be changed to:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returndeserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

• GitHub Issue: [Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist #39010

[github-actions]

⚠️ GitHub issue #39010has been automatically assigned in GitHub to PR creator.

[pitrou]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

[pitrou]

Also:

they can't be read in again with native pyarrow methods without doing extremely ugly and computationally costly workarounds

Please note that from_pylist and to_pylist are quite costly in themselves. Usually you want to avoid these kinds of roundtrips to/from Python objects if you are concerned with performance.

[jonasdedden]

While this is not a bad idea in itself, it seems like the roundtripping concern could be solved more efficiently by making from_pylist accept a list of tuples for map fields.

Let me clarify what this is about. Map fields are already createable with from_pylist by using list of tuples, as I show in the tests I added. Even the code in my initial message can show this. Fundamentally, it's about adding opt-in behaviour to to_pylist to arrive at a functionality one would expect from a Python perspective:

data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
^---------------------------------------------^
this works fine, data will properly encoded in the Arrow way of encoding Maps
^---------^
this will give lists of tuples instead of dicts 

You can use data = [{'x': [('a', 1)]}] here too, this will yield the same RecordBatch. This then of course technically would qualify as a proper "roundtrip", but this is not what this issue is about, it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

Please note that from_pylist and to_pylist are quite costly in themselves.

Yes, but this is part of a very large distributed machine learning setup, where relatively intricate filters applied on deeply nested list/struct/map columns. The compute of the actual machine learning outclasses the compute one has to do to deserialize Python objects by many orders of magnitude.

For pure data queries, we would not use bare Python objects of course.

[pitrou]

it's about deserializing Map Arrow types as the ~"expected" Python equivalent, at least as an opt-in method such as pandas already supports for some couple of years now (shown in the linked Github issue).

I see, thanks. Then, do we want to reuse the same parameter signature as in the Pandas-related PR? I.e., allow either None, "lossy" and "strict", rather than a boolean.

[jonasdedden]

allow either None, "lossy" and "strict", rather than a boolean.

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method? This has to be done because the to_pylist method calls as_py on its member arrays (which can be all possible types), and therefore all array/scalar types have to support this parameter. I did not see any other way to easily implement this. I'm willing to do quick progress here, so if you come up with another idea, let me know.

[pitrou]

Sure, I actually also stumbled across that when I revisited that original Github issue. Before I do that, I'd like to ask whether you're generally fine with adding this new parameter to everyas_py method?

That sounds ok to me. Ideally, to_pylist wouldn't call as_py in a loop (which is going to be quite slow), but that would be a major refactor.

[pitrou]

By the way, we probably want to make the new parameter keyword-only?

[jonasdedden]

I addressed the remarks :) There is some weird error in the "Docs" job, I don't know what this is about.

[pitrou]

Hmm, it looks like some of the CI failures will need #45500 to be merged first

[jonasdedden]

I rebased the branch, now the CI tests seem fine again, I think?

Could we get a approval/review of this? :)

[pitrou]

Thanks @jonded94 ! This looks good on the principle, here are some assorted comments.

[pitrou]

I think the ordering comment is obsolete, as Python dicts are ordered nowadays. Unless the underlying implementation does something weird, ordering should therefore be preserved.

[jonasdedden]

Removed the ordering part, added some explanation of which value survives on duplicate keys.

[pitrou]

Same comment re: ordering

[jonasdedden]

Same as above

[pitrou]

Can we check that a warning is actually emitted? See pytest.warns

[jonasdedden]

Implemented a check for this warning

[pitrou]

Nit: it may be more idiomatic to use the repr here

Suggested change

	+ f"Received '{maps_as_pydicts}'."
	+ f"Received {maps_as_pydicts!r}."

[jonasdedden]

Implemented the suggested change

[pitrou]

I would make this a KeyError. Also, the message should perhaps contain the duplicate key?

[jonasdedden]

Made it a KeyError

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 93045c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-9728f80818

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

+1, will merge if CI is green.

[pitrou]

CI failures are unrelated.

[Linchin]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

[omatthew98]

Just fyi this might cause backward incompatibility issue because the user defined extension types are not expecting maps_as_pydicts as an argument for to_py(). We are seeing this in our prerelease tests:

_________________________ test_json_arrow_record_batch _________________________
def test_json_arrow_record_batch():
data = [
json.dumps(value, sort_keys=True, separators=(",", ":"))
for value in JSON_DATA.values()
]
arr = pa.array(data, type=db_dtypes.JSONArrowType())
batch = pa.RecordBatch.from_arrays([arr], ["json_col"])
sink = pa.BufferOutputStream()
with pa.RecordBatchStreamWriter(sink, batch.schema) as writer:
writer.write_batch(batch)
buf = sink.getvalue()
with pa.ipc.open_stream(buf) as reader:
result = reader.read_all()
json_col = result.column("json_col")
assert isinstance(json_col.type, db_dtypes.JSONArrowType)
> s = json_col.to_pylist()
tests/unit/test_json.py:225: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pyarrow/table.pxi:1380: in pyarrow.lib.ChunkedArray.to_pylist
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
pyarrow/array.pxi:1[67](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:68)7: TypeError
- generated xml file: /home/runner/work/python-db-dtypes-pandas/python-db-dtypes-pandas/unit_prerelease_3.12_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/test_json.py::test_json_arrow_to_pylist - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
FAILED tests/unit/test_json.py::test_json_arrow_record_batch - TypeError: JSONArrowScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
2 failed, 298 passed in 1.[81](https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310#step:5:82)s

(Link: https://github.com/googleapis/python-db-dtypes-pandas/actions/runs/13017353125/job/37736481135?pr=310)

We (Ray Data team) are also running into backward compatibility issues like this in our tests against pyarrow nightly with the same error mentioned here:

[2025-02-25T06:27:20Z] ===================================FAILURES===================================--| [2025-02-25T06:27:20Z] ____________test_convert_to_pyarrow_array_object_ext_type_fallback____________| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] deftest_convert_to_pyarrow_array_object_ext_type_fallback():
| [2025-02-25T06:27:20Z] column_values=create_ragged_ndarray(
| [2025-02-25T06:27:20Z] [
| [2025-02-25T06:27:20Z] "hi",
| [2025-02-25T06:27:20Z] 1,
| [2025-02-25T06:27:20Z] None,
| [2025-02-25T06:27:20Z] [[[[]]]],
| [2025-02-25T06:27:20Z] {"a": [[{"b": 2, "c": UserObj(i=123)}]]},
| [2025-02-25T06:27:20Z] UserObj(i=456),
| [2025-02-25T06:27:20Z] ]
| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z] column_name="py_object_column"| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # First, assert that straightforward conversion into Arrow native types fails| [2025-02-25T06:27:20Z] withpytest.raises(ArrowConversionError) asexc_info:
| [2025-02-25T06:27:20Z] _convert_to_pyarrow_native_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] assert (
| [2025-02-25T06:27:20Z] str(exc_info.value)
| [2025-02-25T06:27:20Z] =="Error converting data to Arrow: ['hi' 1 None list([[[[]]]]) {'a': [[{'b': 2, 'c': UserObj(i=123)}]]}\n UserObj(i=456)]"# noqa: E501| [2025-02-25T06:27:20Z] )
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] # Subsequently, assert that fallback to `ArrowObjectExtensionType` succeeds| [2025-02-25T06:27:20Z] pa_array=convert_to_pyarrow_array(column_values, column_name)
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] >assertpa_array.to_pylist() ==column_values.tolist()
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] python/ray/air/tests/test_arrow.py:121:
| [2025-02-25T06:27:20Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| [2025-02-25T06:27:20Z]
| [2025-02-25T06:27:20Z] > ???
| [2025-02-25T06:27:20Z] ETypeError: as_py() gotanunexpectedkeywordargument'maps_as_pydicts'

[pitrou]

@Linchin@omatthew98 I think the way around this would be to take a **kwargs in your as_py method and then forward it to any nested as_py call (if any).

For example turn this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self):
returnJSONArray._deserialize_json(self.value.as_py() ifself.valueelseNone)

into this:

classJSONArrowScalar(pa.ExtensionScalar):
defas_py(self, **kwargs):
returnJSONArray._deserialize_json(self.value.as_py(**kwargs) ifself.valueelseNone)

[pitrou]

I've updated the PR description, we should remember to call out this potential incompatibility in the release notes for the next version.