[SPARK-18482][SQL] make sure Spark can access the table metadata created by older version of spark by cloud-fan · Pull Request #16003 · apache/spark

cloud-fan · 2016-11-24T14:52:36Z

What changes were proposed in this pull request?

In Spark 2.1, we did a lot of refactor for HiveExternalCatalog and related code path. These refactor may introduce external behavior changes and break backward compatibility. e.g. http://issues.apache.org/jira/browse/SPARK-18464

To avoid future compatibility problems of HiveExternalCatalog, this PR dumps some typical table metadata from tables created by 2.0, and test if they can recognized by current version of Spark.

How was this patch tested?

test only change

…n of spark

cloud-fan · 2016-11-24T14:56:27Z

 }
 }
-
- test("SPARK-17470: support old table that stores table location in storage properties") {


This is a useless test. It was added by #15024, which tried to change the way we store table location in metastore. After code review, #15024 revert that change, but forgot to remove this test.

cloud-fan · 2016-11-24T14:56:43Z

- }
- }
-
- test("SPARK-18464: support old table which doesn't store schema in table properties") {


it's covered by the new test suite

cloud-fan · 2016-11-24T14:57:15Z

cc @yhuai @gatorsmile

SparkQA · 2016-11-24T16:42:21Z

Test build #69134 has finished for PR 16003 at commit 8f8c5da.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveExternalCatalogCompatibilitySuite extends QueryTest with TestHiveSingleton

rxin · 2016-11-24T19:39:49Z

+import org.apache.spark.util.Utils
+
+
+class HiveExternalCatalogCompatibilitySuite extends QueryTest with TestHiveSingleton {


To make the name super long...
HiveExternalCatalogBackwardCompatibilitySuite

SparkQA · 2016-11-25T06:56:07Z

Test build #69145 has finished for PR 16003 at commit 6f785e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveExternalCatalogBackwardCompatibilitySuite extends QueryTest with TestHiveSingleton

rxin · 2016-11-25T18:32:49Z

The main thing I'd add is to add comment explaining what version of Spark would generate those table props.

gatorsmile · 2016-11-26T07:24:43Z

How about Spark 2.1 altering the table metadata created by Spark 2.0?

SparkQA · 2016-11-27T07:12:33Z

Test build #69200 has started for PR 16003 at commit a42b8b9.

SparkQA · 2016-11-27T10:32:29Z

Test build #3440 has finished for PR 16003 at commit a42b8b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-11-27T22:16:48Z

+
+
+ // Raw table metadata that are dumped from tables created by Spark 2.0. Note that, all spark
+ // versions prior to 2.1 would generate same raw table metadata for a specific table.


I briefly checked 1.6. Most of them are the same, but some changes are only available in 2.0. For example, locationUri = Some(defaultTablePath("tbl7") + "-__PLACEHOLDER__"), was added in #13270

gatorsmile · 2016-11-27T22:18:32Z

LGTM except a minor comment. We can address it in a separate PR for checking 1.6.

SparkQA · 2016-11-28T05:40:34Z

Test build #69217 has finished for PR 16003 at commit 117f532.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-11-28T05:45:36Z

Merging in master/branch-2.1.

…ted by older version of spark ## What changes were proposed in this pull request? In Spark 2.1, we did a lot of refactor for `HiveExternalCatalog` and related code path. These refactor may introduce external behavior changes and break backward compatibility. e.g. http://issues.apache.org/jira/browse/SPARK-18464 To avoid future compatibility problems of `HiveExternalCatalog`, this PR dumps some typical table metadata from tables created by 2.0, and test if they can recognized by current version of Spark. ## How was this patch tested? test only change Author: Wenchen Fan <wenchen@databricks.com> Closes#16003 from cloud-fan/test. (cherry picked from commit fc2c13b) Signed-off-by: Reynold Xin <rxin@databricks.com>

…ted by older version of spark ## What changes were proposed in this pull request? In Spark 2.1, we did a lot of refactor for `HiveExternalCatalog` and related code path. These refactor may introduce external behavior changes and break backward compatibility. e.g. http://issues.apache.org/jira/browse/SPARK-18464 To avoid future compatibility problems of `HiveExternalCatalog`, this PR dumps some typical table metadata from tables created by 2.0, and test if they can recognized by current version of Spark. ## How was this patch tested? test only change Author: Wenchen Fan <wenchen@databricks.com> Closesapache#16003 from cloud-fan/test.

make sure Spark can access the table metadata created by older versio…
8f8c5da
…n of spark

cloud-fan commented Nov 24, 2016
View reviewed changes

rxin reviewed Nov 24, 2016
View reviewed changes

address comment
6f785e3

address comments
a42b8b9

gatorsmile reviewed Nov 27, 2016
View reviewed changes

improve comments
117f532

asfgit closed this in fc2c13bNov 28, 2016

cloud-fan mentioned this pull request Jan 18, 2017
[SPARK-19265][SQL] make table relation cache general and does not depend on hive #16621
Closed

		import org.apache.spark.util.Utils


		class HiveExternalCatalogCompatibilitySuite extends QueryTest with TestHiveSingleton {



		// Raw table metadata that are dumped from tables created by Spark 2.0. Note that, all spark
		// versions prior to 2.1 would generate same raw table metadata for a specific table.

Uh oh!

Conversation

cloud-fan commented Nov 24, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

cloud-fanNov 24, 2016

Choose a reason for hiding this comment

Uh oh!

cloud-fanNov 24, 2016

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Nov 24, 2016

Uh oh!

SparkQA commented Nov 24, 2016

Uh oh!

rxinNov 24, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 25, 2016

Uh oh!

rxin commented Nov 25, 2016

Uh oh!

gatorsmile commented Nov 26, 2016

Uh oh!

SparkQA commented Nov 27, 2016

Uh oh!

SparkQA commented Nov 27, 2016

Uh oh!

gatorsmileNov 27, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Nov 27, 2016

Uh oh!

SparkQA commented Nov 28, 2016

Uh oh!

rxin commented Nov 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants