[SPARK-19219][SQL] Fix Parquet log output defaults

[nicklavers]

What changes were proposed in this pull request?

Changing the default parquet logging levels to reflect the changes made in PR #15538, in order to prevent the flood of log messages by default.

How was this patch tested?

Default log output when reading from parquet 1.6 files was compared with and without this change. The change eliminates the extraneous logging and makes the output readable.

[nicklavers]

Link to this JIRA issue: https://issues.apache.org/jira/browse/SPARK-19219
Link to original issue: https://issues.apache.org/jira/browse/SPARK-17993

[srowen]

So the change here is really to turn up the log level for non-test code, not just the test code? it seems possibly reasonable but are there important warnings this would suppress for users?

[nicklavers]

That's possible. We could narrow the scope a bit and instead just set log4j.logger.org.apache.parquet.CorruptStatistics=ERROR log4j.logger.parquet.CorruptStatistics=ERROR,
which also fixes the problem.

[SparkQA]

Test build #71345 has finished for PR 16580 at commit cb80164.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@nicklavers yeah narrowing the scope seems more conservative and desirable. Do the test configs need to be changed similarly to match your newer proposal?

[nicklavers]

I don't see why not

[SparkQA]

Test build #71459 has finished for PR 16580 at commit c7df9ac.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71462 has finished for PR 16580 at commit 6788987.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71460 has finished for PR 16580 at commit cb80164.

• This patch fails from timeout after a configured wait of `250m`.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master