Add way to disables statistics on a per column basis

[asfimport]

When you write dataset with BINARY columns that can be fairly large (several Mbs) you can often end with an OutOfMemory error where you either have to:

 

 - Throw more RAM

 - Increase number of output files

 - Play with Block size

 

Using a fork with increased checks frequency for row group size help but it is not enough. (PR: #470)

 

 

The OutOfMemory error is now caused due to the accumulation of min/max values for those columns for each BlockMetaData.

 

The "parquet.statistics.truncate.length" configuration is of no help because it is applied during the footer serialization whereas the OOM occurs before that.

 

I think it would be nice to have, like for dictionary or bloom filter, a way to disable the statistic on a per-column basis.

 

Could be very useful to lower memory consumption when stats of huge binary column are unnecessary.

 

 

 

Reporter: Anthony Pessy / @panthony

Original Issue Attachments:

• add_config_to_opt-out_of_a_column's_statistics.patch
• NoOpStatistics.java

_{Note: This issue was originally created as PARQUET-1911. Please see the migration documentation for further details.}

[asfimport]

Anthony Pessy / @panthony:
I could successfully pass a job with less than 2.5 Gb of memory (whereas ~7-8Gb was previously needed) using the provided patch.

 

I was not sure of the best approach regarding constructors (& breaking changes), I went with the less modifications possibles but some of them could take ParquetProperties directly instead.

 

 

 

I used a NoOp statistics to avoid copying/comparing huge byte array for each write since it would be dropped anyway.

[asfimport]

Igor Berman:
@panthony can you please attach new files ? your patch contains only changes
in addition, have you tested if working with such data works? i.e. i think there are all kind of assumption regarding referencing statistics at read path(i.e. predicate pushdowns etc)

[asfimport]

Anthony Pessy / @panthony:
[~igor.berman] One of the use case I have is for long term storage of HTML payloads, the dataset contains columns such as url, status_code, fetch_date, document_html sorted by url.

 

I'm using predicate pushdowns filters on URL to quickly find the relevant document and it work well, in this dataset I do not care about keeping statistics for the document_html column which was producing the OOM, especially for websites where all pages where around ~7-8Mb of payload.

 

I also use this feature in many intermediate parquet files I'm writing using Spark and/or HadoopJob that contains similar data & it heavily helped with the memory requirements.

 

I added the new NoOpStatistics file

 

[asfimport]

Xinli Shang / @shangxinli:
@panthony Thanks for working on this! Just FYI that there was an effort to truncate the min/max https://issues.apache.org/jira/browse/PARQUET-1685. It can be enabled with a flag. With that said, your changes are still welcome. Feel free to create a PR if you haven't and I will review it.

[asfimport]

Anthony Pessy / @panthony:
[~shangx@uber.com] Sorry for the late reply it seems I did not receive a notification.

 

The truncation on min/max is not sufficient because I'm having an out of memory way before I'm even close to writing a footer as the truncation takes place when it actually write the footer (If I recall) and it still keep the whole values in memory until then.

 

(The whole idea of min/max against a HTML payload or similar fields does not have much sense anyway).

 

I'll try to see if I can gather the time to suggest a PR as I'd like to stop relying on a forked version.