GH-3163: Reduce memory and time overhead of ParquetRewriterTests

[rahulketch]

Rationale for this change

Reduce the memory overhead and the time taken to run ParquetRewriterTests

What changes are included in this PR?

Reducing number of records from 100000 to 10000

Are these changes tested?

Are there any user-facing changes?

No

Closes #GH-3163

[wgtmac]

This may result in a single page for each column chunk. Could you try following things:

• Make sure more than one page are produced by setting config like parquet.page.value.count.threshold and relevant page size check parameters at https://github.com/apache/parquet-java/blob/master/parquet-hadoop/README.md
• Parameterize numRecord (and page size check config) and by default use 10000?

[rahulketch]

@wgtmac : thanks for your comment. Do you happen to know how I can run a single test in the repository after making my changes? I wanted to run only the ParquetRewriterTest, but have not figured out a good way to achieve that via mvn.

[wgtmac]

cd ~/Projects/parquet-java # replace with your project root directory
cd parquet-hadoop
mvn test -Dtest=org.apache.parquet.hadoop.rewrite.ParquetRewriterTest

[rahulketch]

As far as I understand, the relevant config parameter is parquet.page.row.count.limit and I have changed that to be num_records / 5. Hope it looks good!

[rahulketch]

@wgtmac : Will this line also require a change?

parquet-java/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/rewrite/ParquetRewriterTest.java

Line 928 in d20ed14

.withRowGroupSize(1 * 1024 * 1024)

[wgtmac]

@wgtmac : Will this line also require a change?

parquet-java/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/rewrite/ParquetRewriterTest.java

Line 928 in d20ed14

.withRowGroupSize(1 * 1024 * 1024)

Yes, I think it would be good to test more than one row group.

[rahulketch]

@wgtmac : Will this line also require a change?

parquet-java/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/rewrite/ParquetRewriterTest.java

Line 928 in d20ed14

.withRowGroupSize(1 * 1024 * 1024)

Yes, I think it would be good to test more than one row group.

I verified that currently the test creates more than 1 row group with the current values. Let me know if any more concerns before merging.

[wgtmac]

LGTM Thanks!

cc @ConeyLiu

[ConeyLiu]

+1, thanks for the contribution.