[C++][Parquet] Support LargeListArray

[asfimport]

For now it's not possible to write a pyarrow.Table containing a LargeListArray in parquet. The lines

from pyarrow import parquet
import pyarrow as pa

indices = [1, 2, 3]
indptr = [0, 1, 2, 3]
q = pa.lib.LargeListArray.from_arrays(indptr, indices) 
table = pa.Table.from_arrays([q], names=['no']) 

parquet.write_table(table, '/test')

yields the error 

ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: large_list<item: int64>

 

Reporter: marc abboud
Assignee: Micah Kornfield / @emkornfield

Related issues:

• [Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows (is related to)

_{Note: This issue was originally created as ARROW-7731. Please see the migration documentation for further details.}

[asfimport]

Artem KOZHEVNIKOV / @artemru:
I found another edge case that's maybe linked to this (pyarrow=0.15.1)

import pyarrow as pa
import pyarrow.parquet as pq
l1 = pa.array([list(range(100))] * 10**7, type=pa.list_(pa.int16()))
tt = pa.Table.from_pydict(\{'big': pa.chunked_array([l1]*10)}) # if concat, offset will overflow int32
pq.write_table(tt, '/tmp/test.parquet') # that took a while but worked
tt_reload = pq.read_table('/tmp/test.parquet') # it consumes a huge amount of memory before failing

ArrowInvalid Traceback (most recent call last) <ipython-input-7-bf871e1a4f57> in <module> ----> 1 tt_reload = pq.read_table('/tmp/test.parquet') /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/parquet.py in read_table(source, columns, use_threads, metadata, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size) 1279 buffer_size=buffer_size) 1280 return pf.read(columns=columns, use_threads=use_threads, -> 1281 use_pandas_metadata=use_pandas_metadata) 1282 1283 /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, use_threads, use_pandas_metadata) 1135 table = piece.read(columns=columns, use_threads=use_threads, 1136 partitions=self.partitions, -> 1137 use_pandas_metadata=use_pandas_metadata) 1138 tables.append(table) 1139 /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, use_threads, partitions, file, use_pandas_metadata) 603 table = reader.read_row_group(self.row_group, **options) 604 else: --> 605 table = reader.read(**options) 606 607 if len(self.partition_keys) > 0: /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, use_threads, use_pandas_metadata) 251 columns, use_pandas_metadata=use_pandas_metadata) 252 return self.reader.read_all(column_indices=column_indices, --> 253 use_threads=use_threads) 254 255 def scan_contents(self, columns=None, batch_size=65536): /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetReader.read_all() /opt/conda/envs/model/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() ArrowInvalid: Column 0: Offset invariant failure: 21474837 inconsistent offset for non-null slot: -2147483596<2147483600

Thrown Error is not explicit. I wonder if created parquet file is correct (I've not tried yet to reload it with spark) or it's just a reading by pyarrow that does not support it.

[asfimport]

Wes McKinney / @wesm:
Either way that looks like the wrong exception to throw

[asfimport]

Micah Kornfield / @emkornfield:
Writing should be supported now (as of 0.16.0) please update the bug of you are still seeing issues. Reading isn't.

[asfimport]

Artem KOZHEVNIKOV / @artemru:
yes, I confirm that writing is ok now, reading is still broken !

[asfimport]

Micah Kornfield / @emkornfield:
Supporting reading is tacked in ARROW-1644 and its children.

[asfimport]

Artem KOZHEVNIKOV / @artemru:
thanks a lot 

[asfimport]

Artem KOZHEVNIKOV / @artemru:
but the concat still does not work :

import pyarrow as pa
import pyarrow.parquet as pq
l1 = pa.array([list(range(100))] * 10**7, type=pa.list_(pa.int16()))
tt = pa.Table.from_pydict({'big': pa.chunked_array([l1] * 10)}) # if concat, offset will overflow int32
tt.combine_chunks()

---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
<ipython-input-3-39fcc668ae13> in <module>
----> 1 tt.combine_chunks()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.combine_chunks()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: offset overflow while concatenating arrays

[asfimport]

Wes McKinney / @wesm:
Is there an issue about that, or if not can you create one and link it to this issue?

[asfimport]

Todd Farmer / @toddfarmer:
Transitioning issue from Resolved to Closed to based on resolution field value.