Parquet read fails in certain scenarios for files written by parquet-go

[ArnavBalyan]

Describe the enhancement requested

Parquet-go can write extra entries in the column index for 1 column chunk. This causes Parquet java to fail with java.lang.ArrayIndexOutOfBoundsException when using column indexes.

• Parquet java expects the same number of entries in column index and offset index (one entry per data page).
• When they don't match, the reader seems to be trying to read a non existent page (based on the column stats) and fails. (stacktrace below)

Structure of indexes which lead to the failure:

DEBUG: COLUMN1 ColumnIndex=5 stats
 Entry 0: nullPage=false, nullCount=0, min='', max=''
 Entry 1: nullPage=false, nullCount=0, min='xyz', max='abc'
 Entry 2: nullPage=false, nullCount=0, min='xyz', max='abc'
 Entry 3: nullPage=false, nullCount=0, min='xyz', max='abc'
 Entry 4: nullPage=false, nullCount=0, min='xyz', max='abc'

DEBUG: COLUMN1 OffsetIndex=4 pages
 Entry 0: offset=..., compressedSize=..., firstRowIndex=0
 Entry 1: offset=..., compressedSize=..., firstRowIndex=30
 Entry 2: offset=..., compressedSize=..., firstRowIndex=60
 Entry 3: offset=..., compressedSize=..., firstRowIndex=90

(Here we have 1 additional column index for the same data page)

There could be potentially 2 ways to fix this:

1. Detect inconsistent metadata and do not rely on column index for unsuitable/incomplete metadata. Log a WARN and continue reading the file without index, which will correctly read the file.
2. Clearly throw an exception indicating the missing/unhandled metadata (in this case, the mismatch in number of column/offset index), so user can disable column index or correct upstream.

Stacktrace:

 at org.apache.parquet.internal.column.columnindex.OffsetIndexBuilder$OffsetIndexImpl.getFirstRowIndex(OffsetIndexBuilder.java:66)
 at org.apache.parquet.internal.filter2.columnindex.RowRanges.create(RowRanges.java:144) 
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.applyPredicate(ColumnIndexFilter.java:189)
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.visit(ColumnIndexFilter.java:126)
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.visit(ColumnIndexFilter.java:57) 
 at org.apache.parquet.filter2.predicate.Operators$NotEq.accept(Operators.java:192)
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter$1.visit(ColumnIndexFilter.java:87) 
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter$1.visit(ColumnIndexFilter.java:82) 
 at org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:149) 
 at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.calculateRowRanges(ColumnIndexFilter.java:82)
 at org.apache.parquet.hadoop.ParquetFileReader.getRowRanges(ParquetFileReader.java:1219) 
 at org.apache.parquet.hadoop.ParquetFileReader.getFilteredRecordCount(ParquetFileReader.java:875) 
...

Component(s)

No response

[ArnavBalyan]

cc @gszadovszky just wanted to know your thoughts. Ignoring column index and reading seems to avoid the issue but may silently introduce a dependency on incorrect behaviour from upstream.

[wgtmac]

Is this a known issue of Parquet-go?

cc @zeroshade

[zeroshade]

This is not a known issue that I'm aware of. Importantly, is this github.com/parquet-go/parquet-go? Or is this the Go library for parquet in the arrow-go repo?

If it's the latter, can you share the code that was used so I can fix this in the Go library? If it's the former, we should file an issue on parquet-go

[ArnavBalyan]

Looks like it's one of the several open source implementations of parquet-go. the upstream is buggy, will check thanks for taking a look @wgtmac @zeroshade. I think it may be helpful to add an improved error message when the read fails due to corrupt metadata, will try to improve it thanks!

[gszadovszky]

Related to handling such issues in parquet-java during reading, I would vote on the 1st approach. Statistics are not required to access the data so we should not fail but ignore invalid statistics. We shall log a proper message to make the situation clear.

[ArnavBalyan]

Yes agreed, let me see if there is a clean way to do this