PARQUET-374: Add api to read dictionary from each column chunk for predicate pushdown

[zhenxiao]

No description provided.

[julienledem]

please make sure indentation is consistent with existing code

[julienledem]

Thanks for the contribution.
Comments are mostly about reorganizing code in the places that make more sense.

[zhenxiao]

@julienledem I tested against both PAGE_V1 and PAGE_V2, this heuristics seems working OK, an ideal solution is to use PageEncodingStats, this is a heuristics workaround. I put todo here as a next step

[zhenxiao]

thank you, @julienledem get comments addressed

[julienledem]

Possibly you could check if you can read only the dictionary page instead of the entire column chunk.

[zhenxiao]

@julienledem seems like could not decide the compressed page size until reading the page header, so at this time(before reading page header), do not know the actual size to read. I will leave todo here

[zhenxiao]

thank you @julienledem get comments addressed

[danielcweeks]

@zhenxiao Isn't this name a little misleading? I believe this is better as getCurrentRowGroupDictionaries(...) because it just returns the current row group and doesn't advance.

[zhenxiao]

@danielcweeks my bad, forgot to advance the row group, just get it fixed. Following the readNextRowGroup, call this readNextRowGroupDictionaries

[zhenxiao]

@danielcweeks you are correct. read dictionaries should not advance the row group, only after reading the whole row group does. I get the method name updated

[danielcweeks]

@julienledem Unfortunately, I don't think this API works very well for the uses cases we have right now. The issue @zhenxiao is seeing is that he wants to identify the row groups not filtered prior to processing them. This interface requires that you do both in lock step. The filter api has the same problem in that it basically needs to do up front row group filtering without loading any column data. We would have to be able to advance the dictionary separately from the row group, which would make this interface rather clunky.

Overall, I think the static interface was closer to what we currently need (though I can see how this approach can be used to filter while processing). Maybe we just need to separate out the reading of the dictionary as a utility that gets used by this.

[zhenxiao]

@julienledem @danielcweeks as @danielcweeks explained, the use case is,
read footer, for each row group read blockMetaData and Dictionary for each predicate column chunks, decide whether to read the row group or not based on stats and dictionary, keep the list of to-be-scanned row groups, and then start scanning the necessary row groups.
In this case advance dictionary is separate from advancing row group data.
I am thinking about the following options:

1. advance blockId in both nextRowGroup() and nextRowGroupDictionaries(), while, they should not be called interleaving or together, only one of them could be called in the application
2. have a user level advanceBlock() which will advance blockId, then user could decide when to call nextRowGroup(), when to advanceBlock(), and when to read dictionaries, this could be a little bit too flexible, and existing applications calling nextRowGroup() needs to be updated
3. have a static method in ParquetFileReader, pass BlockMetaData, configuration, path as input parameters
   Agree with @danielcweeks option#3 is better. what do you think?