Skip to content

[SPARK-22279][SQL] Enable convertMetastoreOrc by default - #21186

Closed
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-24112
Closed

[SPARK-22279][SQL] Enable convertMetastoreOrc by default#21186
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-24112

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Apr 27, 2018•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

We reverted spark.sql.hive.convertMetastoreOrc at #20536 because we should not ignore the table-specific compression conf. Now, it's resolved via SPARK-23355.

How was this patch tested?

Pass the Jenkins.

@SparkQA

SparkQA commented Apr 27, 2018

Copy link
Copy Markdown

Test build #89941 has finished for PR 21186 at commit 5383299.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Apr 29, 2018•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

@gatorsmile and @cloud-fan .
Could you review this PR? This is a first try after the reverting (#20536).
If we have more things to do for this, please let me know.

@dongjoon-hyundongjoon-hyun changed the title [SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastore.TableProperty conf[SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastoreTableProperty confMay 1, 2018
@dongjoon-hyun

dongjoon-hyun commented May 1, 2018

Copy link
Copy Markdown
MemberAuthor

Hi, @gatorsmile .
Do you think we need to split this into two separate PRs? If you want, I will split this.

  • SPARK-22279 Enable convertMetastoreOrc by default
  • SPARK-24112 Add spark.sql.hive.convertMetastoreTableProperty for backward compatiblility

@dongjoon-hyun

dongjoon-hyun commented May 1, 2018

Copy link
Copy Markdown
MemberAuthor

Retest this please.

@SparkQA

SparkQA commented May 1, 2018

Copy link
Copy Markdown

Test build #89986 has finished for PR 21186 at commit 5383299.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented May 1, 2018

Copy link
Copy Markdown
MemberAuthor

The failures are irrelevant to this PR.

@dongjoon-hyun

dongjoon-hyun commented May 1, 2018

Copy link
Copy Markdown
MemberAuthor

Retest this please.

@SparkQA

SparkQA commented May 1, 2018

Copy link
Copy Markdown

Test build #89998 has finished for PR 21186 at commit 5383299.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented May 2, 2018

Copy link
Copy Markdown
Contributor

spark.sql.hive.convertMetastoreTableProperty looks unnecessary to me...

@dongjoon-hyun

dongjoon-hyun commented May 2, 2018

Copy link
Copy Markdown
MemberAuthor

Ya. I also thought like that before, @cloud-fan .

Please consider an existing customer environment like the unit test cases. For some Parquet tables having table properties like TBLPROPERTIES (parquet.compression 'NONE'), it was ignored by default before Apache Spark 2.4. After upgrading cluster, Spark will write uncompressed file which is different from Apache Spark 2.3 and old.

Since this is a behavior change, we need to document it and had better provide options for this. We can remove this at Apache Spark 3.0.

@SparkQA

SparkQA commented May 3, 2018

Copy link
Copy Markdown

Test build #90149 has finished for PR 21186 at commit b9ed640.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed May 4, 2018
View reviewed changes
Comment threaddocs/sql-programming-guide.md Outdated

@cloud-fancloud-fanMay 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please polish the migration guide w.r.t. https://issues.apache.org/jira/browse/SPARK-24175

@dongjoon-hyundongjoon-hyunMay 4, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

dongjoon-hyun
dongjoon-hyun commented May 4, 2018
View reviewed changes
Comment threaddocs/sql-programming-guide.md Outdated

@dongjoon-hyundongjoon-hyunMay 4, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan and @gatorsmile . I updated according to the guideline SPARK-24175.

@SparkQA

SparkQA commented May 4, 2018

Copy link
Copy Markdown

Test build #90213 has finished for PR 21186 at commit b746702.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented May 6, 2018

Copy link
Copy Markdown
MemberAuthor

Retest this please.

@SparkQA

SparkQA commented May 6, 2018

Copy link
Copy Markdown

Test build #90263 has finished for PR 21186 at commit b746702.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented May 7, 2018•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

I'll split this into two PRs in order to make it easy to review.

@dongjoon-hyundongjoon-hyun changed the title [SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastoreTableProperty conf[SPARK-22279][SQL] Enable convertMetastoreOrc by defaultMay 7, 2018
@dongjoon-hyun

dongjoon-hyun commented May 7, 2018

Copy link
Copy Markdown
MemberAuthor

To reduce the review scope, convertMetastoreTableProperty goes to #21259 .

@SparkQA

SparkQA commented May 7, 2018

Copy link
Copy Markdown

Test build #90332 has finished for PR 21186 at commit ddd6872.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented May 9, 2018

Copy link
Copy Markdown
Contributor

can you resolve the conflicts?

@dongjoon-hyun

dongjoon-hyun commented May 9, 2018

Copy link
Copy Markdown
MemberAuthor

Sure, it's rebased now.

@SparkQA

SparkQA commented May 9, 2018

Copy link
Copy Markdown

Test build #90417 has finished for PR 21186 at commit 10e8319.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented May 10, 2018

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@dongjoon-hyun

dongjoon-hyun commented May 10, 2018

Copy link
Copy Markdown
MemberAuthor

Thank you, @cloud-fan !

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-24112 branch May 10, 2018 15:44
otterc pushed a commit to linkedin/spark that referenced this pull request Mar 22, 2023
@dongjoon-hyunfli
[SPARK-22279][SQL] Enable convertMetastoreOrc by default
ac59c17
We reverted `spark.sql.hive.convertMetastoreOrc` at apache#20536 because we should not ignore the table-specific compression conf. Now, it's resolved via [SPARK-23355](apache@8aa1d7b).
Pass the Jenkins.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closesapache#21186 from dongjoon-hyun/SPARK-24112.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fancloud-fancloud-fan left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dongjoon-hyun@SparkQA@cloud-fan