[Python] Can not refer to field in a list of structs

[asfimport]

When the dataset has nested sturcts, "list", we can not use pyarrow.field(..) to get the reference of the sub-field of the struct.

 

For example

 

import pyarrow as pa
import pyarrow.dataset as ds
import pandas as pd

schema = pa.schema(
 [
 pa.field(
 "objects",
 pa.list_(
 pa.struct(
 [
 pa.field("name", pa.utf8()),
 pa.field("attr1", pa.float32()),
 pa.field("attr2", pa.int32()),
 ]
 )
 ),
 )
 ]
)

table = pa.Table.from_pandas(
 pd.DataFrame([{"objects": [{"name": "a", "attr1": 5.0, "attr2": 20}]}])
)
print(table)

dataset = ds.dataset(table)
print(dataset)
dataset.scanner(columns=["objects.attr2"]).to_table()

which throws exception:

Traceback (most recent call last):
 File "foo.py", line 31, in <module>
 dataset.scanner(columns=["objects.attr2"]).to_table()
 File "pyarrow/_dataset.pyx", line 298, in pyarrow._dataset.Dataset.scanner
 File "pyarrow/_dataset.pyx", line 2356, in pyarrow._dataset.Scanner.from_dataset
 File "pyarrow/_dataset.pyx", line 2202, in pyarrow._dataset._populate_builder
 File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(objects.attr2) in objects: list<item: struct<attr1: double, attr2: int64, name: string>>
__fragment_index: int32
__batch_index: int32
__last_in_fragment: bool
__filename: string

Reporter: Lei (Eddy) Xu

Related issues:

• [Python] parquet.read_table nested fields in columns does not work for use_legacy_dataset=False (relates to)

_{Note: This issue was originally created as ARROW-17540. Please see the migration documentation for further details.}

[asfimport]

Miles Granger / @milesgranger:
I think this is a duplicate of ARROW-14596 ? (That will also solve providing dotted path when use_legacy_dataset=False)

[asfimport]

Miles Granger / @milesgranger:
Should also mention, that if you are only after a single list element, you can do the following, albeit ugly, bit of code here. Until it's properly fixed.

dataset.to_table(columns={
 'attr2': pc.struct_field(
 pc.list_element(ds.field("objects"), ds.scalar(0)), 
 [1])
 }
)