[SPARK-22158][SQL] convertMetastore should not ignore table property by dongjoon-hyun · Pull Request #19382 · apache/spark

dongjoon-hyun · 2017-09-28T17:29:29Z

What changes were proposed in this pull request?

From the beginning, convertMetastoreOrc ignores table properties and use an empty map instead. This PR fixes that. For the diff, please see this. convertMetastoreParquet also ignore.

valoptions=Map[String, String]()

How was this patch tested?

Pass the Jenkins with an updated test suite.

…ties

SparkQA · 2017-09-28T19:23:09Z

Test build #82283 has finished for PR 19382 at commit 09c621b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-09-29T07:05:24Z

+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')


It sounds like we have the same issue for parquet?

Thank you for review, @gatorsmile . I'll check that, too.

dongjoon-hyun · 2017-09-29T17:12:03Z

 val serde = relation.tableMeta.storage.serde.getOrElse("").toLowerCase(Locale.ROOT)
 if (serde.contains("parquet")) {
- val options = Map(ParquetOptions.MERGE_SCHEMA ->
+ val options = relation.tableMeta.storage.properties + (ParquetOptions.MERGE_SCHEMA ->


Hi, @gatorsmile . It's the same in the Paquet. Without this, the test case for parquet fails.

dongjoon-hyun · 2017-09-29T17:12:29Z

- |LOCATION '${path.toURI}'
- """.stripMargin)
+|CREATE TABLE t(id int) USING hive
+|OPTIONS(fileFormat 'parquet', compression 'gzip')


parquet use gzip.

Could you combine these two test cases?

Unfortunately, it's not feasible. First of all,
Old ORC test case covers convertMetastoreOrc=false only. So, I extended it to cover both.
New Parquet test case covers `convertMetastoreParquet=true (default) only because Parquet has another bug when convertMetastoreParquet=false.
Also, the compression codec check is quite different between ORC and Parquet. Please see line 1510.

Still can create a common helper function, right?

Try your best to reduce duplicate codes.

dongjoon-hyun · 2017-09-29T17:14:22Z

+ sparkContext.hadoopConfiguration,
+ new Path(maybeParquetFile.get.getPath),
+ NO_FILTER)
+ assert("GZIP" === footer.getBlocks.get(0).getColumns().get(0).getCodec.toString)


It's for getting compression codec.

dongjoon-hyun · 2017-09-29T17:31:48Z

@gatorsmile .
Whiling testing, I noticed that Parquet ignores table properties in case of convertMetastoreParquet=false, too. For that case, we need to proceed in another JIRA issue. #19218 may cover that, too.

SparkQA · 2017-09-29T18:34:45Z

Test build #82324 has finished for PR 19382 at commit d218c98.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-09-29T18:37:25Z

It fails due to flume dependency.

[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.spark#spark-streaming-flume_2.11;2.3.0-SNAPSHOT: not found

dongjoon-hyun · 2017-09-29T18:37:55Z

Retest this please

dongjoon-hyun · 2017-09-29T19:09:25Z

Retest this please.

dongjoon-hyun · 2017-09-29T19:10:31Z

I retriggered because I noticed that master branch is fixed after the previous triggering.

SparkQA · 2017-09-29T19:57:59Z

Test build #82327 has finished for PR 19382 at commit d218c98.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-29T21:22:24Z

Test build #82332 has finished for PR 19382 at commit d218c98.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-10-02T18:50:00Z

Retest this please

SparkQA · 2017-10-02T20:23:23Z

Test build #82391 has finished for PR 19382 at commit 452a003.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-10-02T21:07:10Z

Test build #82395 has finished for PR 19382 at commit 17ccf9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-10-02T21:59:47Z

LGTM

Thanks! Merged to master.

gatorsmile · 2017-10-02T22:01:15Z

Please submit a separate PR to 2.2. Thanks!

dongjoon-hyun · 2017-10-03T02:45:58Z

Thank you, @gatorsmile . I'll.

[SPARK-22158][SQL] convertMetastoreOrc should not ignore table proper…
09c621b
…ties

gatorsmile reviewed Sep 29, 2017
View reviewed changes

fix parquet, too.
144868e

dongjoon-hyun changed the title ~~[SPARK-22158][SQL] convertMetastoreOrc should not ignore table property~~[SPARK-22158][SQL] convertMetastoreOrc/Parquet should not ignore table propertySep 29, 2017

move test case position for review.
d218c98

dongjoon-hyun commented Sep 29, 2017
View reviewed changes

dongjoon-hyun added 2 commits October 2, 2017 11:23

Update test cases.
452a003

fix
17ccf9a

dongjoon-hyun changed the title ~~[SPARK-22158][SQL] convertMetastoreOrc/Parquet should not ignore table property~~[SPARK-22158][SQL] convertMetastore should not ignore table propertyOct 2, 2017

asfgit closed this in e5431f2Oct 2, 2017

dongjoon-hyun deleted the SPARK-22158 branch October 3, 2017 02:46

dongjoon-hyun mentioned this pull request Oct 3, 2017
[SPARK-22158][SQL][BRANCH-2.2] convertMetastore should not ignore table property #19417
Closed

gatorsmile mentioned this pull request Feb 7, 2018
[SPARK-23355][SQL] convertMetastore should not ignore table properties #20522
Closed

Uh oh!

Conversation

dongjoon-hyun commented Sep 28, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 28, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Sep 29, 2017

Uh oh!

SparkQA commented Sep 29, 2017

Uh oh!

dongjoon-hyun commented Sep 29, 2017

Uh oh!

dongjoon-hyun commented Sep 29, 2017

Uh oh!

dongjoon-hyun commented Sep 29, 2017

Uh oh!

dongjoon-hyun commented Sep 29, 2017

Uh oh!

SparkQA commented Sep 29, 2017

Uh oh!

SparkQA commented Sep 29, 2017

Uh oh!

dongjoon-hyun commented Oct 2, 2017

Uh oh!

SparkQA commented Oct 2, 2017

Uh oh!

SparkQA commented Oct 2, 2017

Uh oh!

gatorsmile commented Oct 2, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Oct 2, 2017

Uh oh!

dongjoon-hyun commented Oct 3, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dongjoon-hyun commented Sep 28, 2017•
edited
Loading

gatorsmile commented Oct 2, 2017•
edited
Loading