[SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalog by scwf · Pull Request #4062 · apache/spark

scwf · 2015-01-15T14:17:14Z

Follow up for #3941 and #3960, in this PR
1 enable ddl table name having dot
2 enable hivecontext related api support dot table name with tableIdentifier
3 remove QualifiedTableName which do not match the other API, using tableIdentifier instead.

SparkQA · 2015-01-15T14:22:36Z

Test build #25602 has started for PR 4062 at commit e0d1960.

This patch merges cleanly.

SparkQA · 2015-01-15T15:29:43Z

Test build #25602 has finished for PR 4062 at commit e0d1960.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-15T15:29:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25602/
Test PASSed.

marmbrus · 2015-01-16T21:47:27Z

Why hardcode default here? When a database isn't specified its up to the catalog to use the current one, where as this will ignore any USE statements that have been run.

@marmbrus yes, this just inherited from the old version. Now invalidateTable do not support database as you suggest, so here is a todo in line 102, how about make another PR to fix this? or do it in this PR?

@marmbrus I will make a new PR to fix the database issue after this in.

scwf · 2015-01-20T14:59:30Z

@marmbrus, any other comments here?

scwf · 2015-01-22T15:44:33Z

ping @marmbrus

SparkQA · 2015-01-30T01:22:51Z

Test build #26359 has started for PR 4062 at commit 8857756.

This patch merges cleanly.

SparkQA · 2015-01-30T02:28:33Z

Test build #26359 has finished for PR 4062 at commit 8857756.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-30T02:28:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26359/
Test PASSed.

scwf · 2015-01-30T03:23:06Z

ping @marmbrus, is this ok to go?

scwf · 2015-02-02T03:49:57Z

ping

SparkQA · 2015-02-04T03:12:49Z

Test build #26708 has started for PR 4062 at commit fcfa126.

This patch merges cleanly.

SparkQA · 2015-02-04T03:13:44Z

Test build #26708 has finished for PR 4062 at commit fcfa126.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T03:13:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26708/
Test FAILed.

SparkQA · 2015-02-04T03:52:37Z

Test build #26715 has started for PR 4062 at commit 057d23e.

This patch merges cleanly.

SparkQA · 2015-02-04T05:03:46Z

Test build #26715 has finished for PR 4062 at commit 057d23e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T05:03:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26715/
Test PASSed.

marmbrus · 2015-03-18T02:24:06Z

Okay, sorry this didn't make it into 1.3. Mind bringing it up to date?

scwf · 2015-03-19T09:29:23Z

yeah later back to update this

liancheng · 2015-04-15T17:09:06Z

Sorry... Would you mind to rebase this again? Thanks!

SparkQA · 2015-04-22T11:43:41Z

Test build #30753 has started for PR 4062 at commit e63c4ed.

SparkQA · 2015-04-22T11:47:43Z

Test build #30753 has finished for PR 4062 at commit e63c4ed.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T11:47:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30753/
Test FAILed.

SparkQA · 2015-04-22T12:28:40Z

Test build #30755 has started for PR 4062 at commit 4a318f3.

SparkQA · 2015-04-22T14:06:35Z

Test build #30755 has finished for PR 4062 at commit 4a318f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T14:06:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30755/
Test PASSed.

SparkQA · 2015-04-22T16:28:42Z

Test build #30766 has started for PR 4062 at commit e437e89.

scwf · 2015-04-22T16:34:21Z

Updated and added test cases.

SparkQA · 2015-04-22T18:07:20Z

Test build #30766 has finished for PR 4062 at commit e437e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T18:07:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30766/
Test PASSed.

scwf · 2015-04-22T23:43:09Z

To summarize this:
1 enable ddl table name having dot
2 enable HiveContext/DataFrame/SQLContext related api support dot table name with tableIdentifier
3 remove QualifiedTableName which do not match the other API from HiveMetastoreCatalog, using tableIdentifier instead.
4 tablename x.y.z.....m.n indicate that m is the db name and n is the table name

scwf · 2015-04-22T23:44:05Z

/cc @marmbrus @liancheng

scwf · 2015-04-22T23:47:37Z

should not rename this method

scwf · 2015-04-25T01:45:43Z

Jenkins, retest this please

scwf · 2015-04-27T06:26:37Z

@liancheng , rebased and updated, any comments here?

scwf · 2015-05-03T01:27:00Z

Jenkins, retest this please

AmplabJenkins · 2015-05-03T01:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-03T01:27:20Z

Merged build started.

SparkQA · 2015-05-03T01:27:42Z

Test build #31680 has started for PR 4062 at commit 7c65386.

SparkQA · 2015-05-03T03:14:42Z

Test build #31680 has finished for PR 4062 at commit 7c65386.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T03:14:47Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-03T03:14:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31680/
Test PASSed.

scwf · 2015-05-06T21:53:22Z

ping

scwf added 5 commits January 14, 2015 22:40

use tableIdentifier in metastore
fcab9b0

using tableidentfier when create table in hive
111331c

using tableidentifier in ddl also
481ba95

Merge branch 'tableIdentifier' into createtable
3c1fcbb

fix conflicts
e0d1960

marmbrus reviewed Jan 16, 2015
View reviewed changes

This was referenced Jan 16, 2015
[SPARK-5240][SQL] Adding createDataSourceTable interface to Catalog #4036
Closed
[SPARK-5251][SQL] Using tableIdentifier in hive metastore #4045
Closed

scwf changed the title ~~[SPARK-4943][SQL] Allow table name having dot for db/catalog~~[SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalogJan 17, 2015

resolve conflicts
8857756

conflicts fix
fcfa126

style fix
057d23e

fix conflicts
4a318f3

scwf force-pushed the createtable branch from e63c4ed to 4a318f3Compare April 22, 2015 12:23

scwf added 2 commits April 22, 2015 22:27

improvement
a7139fa

added test cases
e437e89

scwf reviewed Apr 22, 2015
View reviewed changes

hiveTableFilePath -> hiveDefaultTableFilePath
7c65386

scwf closed this Jul 1, 2015

Uh oh!

Conversation

scwf commented Jan 15, 2015

Uh oh!

SparkQA commented Jan 15, 2015

Uh oh!

SparkQA commented Jan 15, 2015

Uh oh!

AmplabJenkins commented Jan 15, 2015

Uh oh!

marmbrusJan 16, 2015

Choose a reason for hiding this comment

Uh oh!

scwfJan 16, 2015

Choose a reason for hiding this comment

Uh oh!

scwfJan 17, 2015

Choose a reason for hiding this comment

Uh oh!

scwf commented Jan 20, 2015

Uh oh!

scwf commented Jan 22, 2015

Uh oh!

SparkQA commented Jan 30, 2015

Uh oh!

SparkQA commented Jan 30, 2015

Uh oh!

AmplabJenkins commented Jan 30, 2015

Uh oh!

scwf commented Jan 30, 2015

Uh oh!

scwf commented Feb 2, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

marmbrus commented Mar 18, 2015

Uh oh!

scwf commented Mar 19, 2015

Uh oh!

liancheng commented Apr 15, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

scwfApr 22, 2015

Choose a reason for hiding this comment

Uh oh!

scwf commented Apr 25, 2015

Uh oh!