Format: Add a flag when min/max are truncated

[asfimport]

PARQUET-372 drops page and column chunk stats when values are larger than 4k to avoid storing very large values in page headers and the file footer. An alternative approach is to truncate the values, which would still allow filtering on page stats. The problem with truncating values is that the value in stats may not be the true min or max so engines that use these values as the result of aggregations like min(col) would return incorrect data. We should consider adding metadata to allow truncating values for filtering that captures the fact that the values have been modified.

Reporter: Ryan Blue / @rdblue

Related issues:

• Parquet stats can have awkwardly large values (is related to)

_{Note: This issue was originally created as PARQUET-411. Please see the migration documentation for further details.}

[asfimport]

Raunaq Morarka / @raunaqmorarka:
I believe this issue is addressed by the changes done for https://issues.apache.org/jira/browse/PARQUET-2352

cc: @wgtmac  

[alamb]

Agreed -- thank you @raunaqmorarka -- closing this as the spec now allows for this

Specifically

parquet-format/src/main/thrift/parquet.thrift

Lines 295 to 307 in 9fd57b5

	* Lower and upper bound values for the column, determined by its ColumnOrder.
	*
	* These may be the actual minimum and maximum values found on a page or column
	* chunk, but can also be (more compact) values that do not exist on a page or
	* column chunk. For example, instead of storing "Blart Versenwald III", a writer
	* may set min_value="B", max_value="C". Such more compact values must still be
	* valid values within the column's logical type.
	*
	* Values are encoded using PLAIN encoding, except that variable-length byte
	* arrays do not include a length prefix.
	*/
	5: optional binary max_value;
	6: optional binary min_value;

and

parquet-format/PageIndex.md

Line 85 in 9fd57b5

`max_values[i]="C"`. This allows writers to truncate large values and writers