[SPARK-20958][SQL] Roll back parquet-mr 1.8.2 to 1.8.1

[liancheng]

What changes were proposed in this pull request?

This PR reverts PR #16751, #16791, #16817, #16756, and part of #16795 to roll back parquet-mr 1.8.2 to 1.8.1 to escape from a dependency hell caused by avro 1.7.7 and 1.8.1.

How was this patch tested?

Existing tests.

[dongjoon-hyun]

Sorry for this trouble. @liancheng .
Could you revert #16751 , too?
That was the real upgrade to 1.8.2.

[dongjoon-hyun]

Oh, it's already there in the code. Please just update the PR description.

[viirya]

@liancheng After upgrading to Parquet 1.8.2, #16756 is another PR merged to remove a workaround for old Parquet issue. We should revert it too if rolling back parquet-mr 1.8.1.

[SparkQA]

Test build #77662 has finished for PR 18181 at commit cf461d0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

@dongjoon-hyun I already reverted PR #16751 manually but forgot to mention it in the PR description.

[liancheng]

@viirya Thanks for reminding! I'm reverting that one.

[SparkQA]

Test build #77669 has started for PR 18181 at commit c956201.

[liancheng]

Unfortunately, rolling back parquet-mr to 1.8.1 brings back PARQUET-389, which breaks multiple test cases involving schema evolution (add a new column to a Parquet table and filter on that column).

Trying to figure out a workaround for this but haven't got any luck yet.

cc @marmbrus

[liancheng]

retest this please

[SparkQA]

Test build #77673 has finished for PR 18181 at commit c956201.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Also, cc @rdblue and @mallman .

[rdblue]

-1, with comments on the JIRA issue. I think it is better to include the Parquet fixes in 1.8.2 since Parquet doesn't pull in Avro 1.8.1 - that happens when users declare their own dependency on parquet-avro.

[mallman]

I can't speak to Parquet 1.8.x anymore. We use Parquet 1.9.0 plus a patch for https://issues.apache.org/jira/browse/PARQUET-783 and have had no problems.

[rdblue]

It sounds like we should plan on a 1.8.3 and a 1.9.1 soon in the Parquet community. I'll start this up.

…

On Fri, Jun 2, 2017 at 9:47 AM, Michael Allman ***@***.***> wrote: I can't speak to Parquet 1.8.x anymore. We use Parquet 1.9.0 plus a patch for https://issues.apache.org/jira/browse/PARQUET-783 and have had no problems. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18181 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFXa0FiKB143yePbSsis2Dq-yGOwRLJks5sADy7gaJpZM4Nttfk> .

-- Ryan Blue

[liancheng]

@rdblue@mallman Thanks for the comments! As mentioned in the JIRA ticket, we've decided to preserve parquet-mr 1.8.2. Instead, we'll add a release notes entry to suggest using parquet-avro 1.8.1 instead of 1.8.2 to avoid the Avro dependency conflict.

I'm closing this PR.