Describe the enhancement requested
In some compute engines, like Spark, the Parquet row group is the minimally splittable unit for scanning tasks. Currently, only the row group size is configurable (via parquet.block.size) when writing Parquet files. In some cases, especially for fewer columns with many duplicated values, one row group may have tons of records, thus causes extremely bad performance on downstream Spark queries.
I propose to make the row count limit for each row group configurable. ORC-1172 also has a similar configuration.
Component(s)
Core
Describe the enhancement requested
In some compute engines, like Spark, the Parquet row group is the minimally splittable unit for scanning tasks. Currently, only the row group size is configurable (via
parquet.block.size) when writing Parquet files. In some cases, especially for fewer columns with many duplicated values, one row group may have tons of records, thus causes extremely bad performance on downstream Spark queries.I propose to make the row count limit for each row group configurable. ORC-1172 also has a similar configuration.
Component(s)
Core