[Python] Reconstruct MapArray from Arrays without loss of nulls

[ianmcook]

Describe the bug, including details regarding any error messages, version, and platform.

I have a MapArray created like this:

table = pa.table(
 {"m": [{"a": 1}, {}, None, {"b": None}]},
 schema=pa.schema([pa.field("m", pa.map_(pa.string(), pa.int64()), True)])
)
m = table["m"].chunks[0]

Notice that the value at position 2 is null:

m.to_pandas()
## 0 [(a, 1.0)]
## 1 []
## 2 None
## 3 [(b, None)]
## dtype: object

m.is_null()
## <pyarrow.lib.BooleanArray object at 0x1637cb220>
## [
## false,
## false,
## true,
## false
## ]

I am trying to reconstruct the MapArray from its constituent arrays, but when I do this, it always loses the null:

new = pa.MapArray.from_arrays(m.offsets, m.keys, m.items, m.type)

new.to_pandas()
## 0 [(a, 1.0)]
## 1 []
## 2 []
## 3 [(b, None)]
## dtype: object

new.is_null()
## <pyarrow.lib.BooleanArray object at 0x1637cb220>
## [
## false,
## false,
## false,
## false
## ]

Wrapping pa.array(..., mask=m.is_null()) around it does not help either.

Is there any way to reconstruct the MapArray and keep the null?

Component(s)

Python

[AlenkaF]

From a quick look I would say the null element of the parent MapArray should be determined by the offsets given as an input (with a repeated value). It looks like it doesn’t create a validity bitmap in this case but instead inserts an empty list - I would say this is a bug.

[jorisvandenbossche]

It looks like it doesn’t create a validity bitmap in this case but instead inserts an empty list - I would say this is a bug.

I don't think that's a bug, because an empty map is a valid element for a map array as well.

There is just not enough information in just the offsets/keys/values arrays. It needs additional information, which can be provided either as a separate mask or as nulls in the offsets.

For example:

In [17]: pa.MapArray.from_arrays(pa.array([0, 1, None, 1, 2], pa.int32()), m.keys, m.items, m.type).to_pandas()
Out[17]: 
0 [(a, 1.0)]
1 []
2 None
3 [(b, None)]
dtype: object

(I don't find this very obvious, though .., like where you exactly need to put the None)

The better option would be a mask keyword, so you could pass the result of m.is_null() to it. We have been adding a mask keyword to most other from_arrays methods (eg pa.ListArray.from_arrays has one, so you could take the workaround of first creating a list array and then convert that to a map), but it seems this is still missing in MapArray.from_arrays.

[AlenkaF]

Aha, that makes sense! Thank you for the detailed example Joris.

[ianmcook]

Thanks. I agree that adding a mask argument to pa.MapArray.from_arrays() would be ideal.

Then in the above example we could do

new = pa.MapArray.from_arrays(
 offsets=m.offsets,
 keys=m.keys,
 items=m.items,
 type=m.type,
 mask=m.is_null()
)

Or maybe mask could also be an attribute of MapArray so we could pass mask=m.mask instead of mask=m.is_null().

This would help solve the issue in SPARK-48302 which arose in apache/spark#46529.

[AlenkaF]

I will look into adding a mask argument to MapArray.from_arrays. It seems there will be some C++ changes needed but nothing too complicated.

Will look into adding a mask attribute later also.

[ianmcook]

Relatedly: I noticed that DictionaryArray.from_arrays() takes a mask argument, but it is not implemented. However, this is not so important, because null values in the dictionary indices are preserved:

import pyarrow as pa
import pyarrow.compute as pc

d = pc.dictionary_encode(pa.array([1,1,2,None,3]))

pc.dictionary_decode(pa.DictionaryArray.from_arrays(d.indices, d.dictionary))
## <pyarrow.lib.Int64Array object at 0x12b38b0a0>
## [
## 1,
## 1,
## 2,
## null,
## 3
## ]

[jorisvandenbossche]

Issue resolved by pull request 41757
#41757

[ianmcook]

I opened a PR to use this in PySpark: apache/spark#46837