[SPARK-17659] [SQL] Partitioned View is Not Supported By SHOW CREATE TABLE by gatorsmile · Pull Request #15233 · apache/spark

gatorsmile · 2016-09-25T03:35:00Z

What changes were proposed in this pull request?

Partitioned View is not supported by SPARK SQL. For Hive partitioned view, SHOW CREATE TABLE is unable to generate the right DDL. Thus, SHOW CREATE TABLE should not support it like the other Hive-only features. This PR is to issue an exception when detecting the view is a partitioned view.

How was this patch tested?

Added a test case

SparkQA · 2016-09-25T05:35:09Z

Test build #65874 has finished for PR 15233 at commit c6d3acd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-09-25T07:12:59Z

cc @cloud-fan @yhuai

gatorsmile · 2016-09-25T07:46:18Z

When the target view is a partitioned view, ALTER VIEW AS SELECT and CREATE OR REPLACE VIEW should issue an error too.

cloud-fan · 2016-09-25T07:58:27Z

 }

+ if (h.getTableType == HiveTableType.VIRTUAL_VIEW && partCols.nonEmpty) {
+ unsupportedFeatures += "partitioned view"


Can we read partitioned view in Spark SQL? What does partition mean for a view?

Yeah, partitioned view is a partition-aware view. Users can add or drop partition after creation. For more details, below is the Hive design doc:
https://cwiki.apache.org/confluence/display/Hive/PartitionedViews
Let me try whether reading partitioned view is partition aware in Spark SQL.
Thanks!

After digging it deeper and deeper, I am really doubting the initial motivation of partitioned views makes sense...
First, see the Hive design link: https://cwiki.apache.org/confluence/display/Hive/ViewDev
Update 30-Dec-2009: Prasad pointed out that even without supporting materialized views, it may be necessary to provide users with metadata about data dependencies between views and underlying table partitions so that users can avoid seeing inconsistent results during the window when not all partitions have been refreshed with the latest data. One option is to attempt to derive this information automatically (using an overconservative guess in cases where the dependency analysis can't be made smart enough); another is to allow view creators to declare the dependency rules in some fashion as part of the view definition. Based on a design review meeting, we will probably go with the automatic analysis approach once dependency tracking is implemented. The analysis will be performed on-demand, perhaps as part of describing the view or submitting a query job against it. Until this becomes available, users may be able to do their own analysis either via empirical lineage tools or via view->table dependency tracking metadata once it is implemented. See HIVE-1079.
Update 1-Feb-2011: For the latest on this, see PartitionedViews.
Basically, this feature just affects the metadata of views. It does not affect the query execution.
To add the partition info into the views, users have to manually issue the SQL:
ALTERVIEW view_name ADD [IF NOT EXISTS] partition_spec partition_spec ... ALTERVIEW view_name DROP [IF EXISTS] partition_spec, partition_spec, ...
I read the code changes and test cases in the Hive JIRA: https://issues.apache.org/jira/browse/HIVE-1079. I think we do not need to worry about this Hive-specific feature. The usage scenario is very limited. Maybe the code changes in the existing PR is enough.
If you think we should support it, we might also need the code changes in SHOW PARTITIONS and DESC table PARTITONS. Then, we need to change the fromHivePartition function, because getSD will be NULL for partitioned views; otherwise, we will get a NullPointerException.

cloud-fan · 2016-11-09T00:45:45Z

Sorry I missed this PR. it's straightforward and makes sense to me, can you reopen it?

gatorsmile · 2016-11-09T03:51:02Z

Sure, let me reopen it. : )

gatorsmile · 2016-11-09T03:51:11Z

retest this please

SparkQA · 2016-11-09T05:52:09Z

Test build #68386 has finished for PR 15233 at commit c6d3acd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-11-09T06:59:05Z

LGTM

…ABLE ### What changes were proposed in this pull request? `Partitioned View` is not supported by SPARK SQL. For Hive partitioned view, SHOW CREATE TABLE is unable to generate the right DDL. Thus, SHOW CREATE TABLE should not support it like the other Hive-only features. This PR is to issue an exception when detecting the view is a partitioned view. ### How was this patch tested? Added a test case Author: gatorsmile <gatorsmile@gmail.com> Closes#15233 from gatorsmile/partitionedView. (cherry picked from commit e256392) Signed-off-by: gatorsmile <gatorsmile@gmail.com>

gatorsmile · 2016-11-09T08:13:58Z

Merging to master/2.1. Thanks!

rxin · 2016-11-09T18:00:52Z

What is partitioned view?

cloud-fan · 2016-11-10T01:30:47Z

it's a hive concept: https://cwiki.apache.org/confluence/display/Hive/PartitionedViews

We don't support it and probably don't need it.

rxin · 2016-11-10T01:34:07Z

Wow that's pretty crazy.

gatorsmile · 2016-11-10T08:34:31Z

Yeah, the listed use cases are strange. Not sure why they implemented such a feature. Maybe some features depend on it? I did a search. This feature is not officially documented and Hive users also are confused about it.

…ABLE ### What changes were proposed in this pull request? `Partitioned View` is not supported by SPARK SQL. For Hive partitioned view, SHOW CREATE TABLE is unable to generate the right DDL. Thus, SHOW CREATE TABLE should not support it like the other Hive-only features. This PR is to issue an exception when detecting the view is a partitioned view. ### How was this patch tested? Added a test case Author: gatorsmile <gatorsmile@gmail.com> Closesapache#15233 from gatorsmile/partitionedView.

fix.
c6d3acd

gatorsmile mentioned this pull request Sep 25, 2016
[SPARK-17612][SQL] Support DESCRIBE table PARTITION SQL syntax #15168
Closed

gatorsmile changed the title ~~[SPARK-17659] [SQL] Partitioned View is Not Supported In SHOW CREATE TABLE~~[SPARK-17659] [SQL] Partitioned View is Not Supported By SHOW CREATE TABLESep 25, 2016

cloud-fan reviewed Sep 25, 2016
View reviewed changes

gatorsmile closed this Nov 7, 2016

gatorsmile reopened this Nov 9, 2016

asfgit closed this in e256392Nov 9, 2016

Uh oh!

Conversation

gatorsmile commented Sep 25, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 25, 2016

Uh oh!

gatorsmile commented Sep 25, 2016

Uh oh!

gatorsmile commented Sep 25, 2016

Uh oh!

cloud-fanSep 25, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmileSep 25, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmileSep 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Nov 9, 2016

Uh oh!

gatorsmile commented Nov 9, 2016

Uh oh!

gatorsmile commented Nov 9, 2016

Uh oh!

SparkQA commented Nov 9, 2016

Uh oh!

cloud-fan commented Nov 9, 2016

Uh oh!

gatorsmile commented Nov 9, 2016

Uh oh!

rxin commented Nov 9, 2016

Uh oh!

cloud-fan commented Nov 10, 2016

Uh oh!

rxin commented Nov 10, 2016

Uh oh!

gatorsmile commented Nov 10, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gatorsmileSep 26, 2016•
edited
Loading