Describe the enhancement requested
When reading a Parquet leaf column as Arrow, we presize the Arrow builder so as to avoid spurious reallocations during incremental Parquet decoding calls.
However, the Reserve method on RecordReader will only properly reserve values for non-FLBA non-BYTE_ARRAY physical types.
The result is that, on some of our micro-benchmarks, we spend a significant amount of time reallocating data on the ArrayBuilder. For example, here is a flamegraph of BM_ReadColumnPlain<false,Float16LogicalType>/null_probability:-1:
Component(s)
C++, Parquet
Describe the enhancement requested
When reading a Parquet leaf column as Arrow, we presize the Arrow builder so as to avoid spurious reallocations during incremental Parquet decoding calls.
However, the
Reservemethod onRecordReaderwill only properly reserve values for non-FLBA non-BYTE_ARRAY physical types.The result is that, on some of our micro-benchmarks, we spend a significant amount of time reallocating data on the ArrayBuilder. For example, here is a flamegraph of
BM_ReadColumnPlain<false,Float16LogicalType>/null_probability:-1:Component(s)
C++, Parquet