Skip to content

[SPARK-21785][SQL]Support create table from a parquet file schema - #19434

Closed
jinshen-cn wants to merge 3 commits into
apache:masterfrom
jacshen:master
Closed

[SPARK-21785][SQL]Support create table from a parquet file schema#19434
jinshen-cn wants to merge 3 commits into
apache:masterfrom
jacshen:master

Conversation

@jinshen-cn

@jinshen-cnjinshen-cn commented Oct 5, 2017

Copy link
Copy Markdown

Support create table from a parquet file schema

As described in jira:

CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE'PARQUET''/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
'/user/test/def/'; 

this is a very ugly fix and I would like someone to help to review and refine.
and it only supports create hive table with schema from parquet file.

Tested by test case and tested about build the runnable distribution

test("create table like parquet") {
valf= getClass.getClassLoader.
getResource("test-data/dec-in-fixed-len.parquet").getPath
valv1=""" |create table if not exists db1.table1 like 'parquet'""".stripMargin.concat("'"+ f +"'").concat(
""" |stored as sequencefile |location '/tmp/table1'""".stripMargin
)
val (desc, allowExisting) = extractTableDesc(v1)
assert(allowExisting)
assert(desc.identifier.database ==Some("db1"))
assert(desc.identifier.table =="table1")
assert(desc.tableType ==CatalogTableType.EXTERNAL)
assert(desc.schema ==newStructType()
.add("fixed_len_dec", "decimal(10,2)"))
assert(desc.bucketSpec.isEmpty)
assert(desc.viewText.isEmpty)
assert(desc.viewDefaultDatabase.isEmpty)
assert(desc.viewQueryColumnNames.isEmpty)
assert(desc.storage.locationUri ==Some(newURI("/tmp/table1")))
assert(desc.storage.inputFormat ==Some("org.apache.hadoop.mapred.SequenceFileInputFormat"))
assert(desc.storage.outputFormat ==Some("org.apache.hadoop.mapred.SequenceFileOutputFormat"))
assert(desc.storage.serde ==Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
}

@maropu

Copy link
Copy Markdown
Member

@crazyjacky Can you close this for now cuz it's not active for a long time?

@AmplabJenkins

Copy link
Copy Markdown

Can one of the admins verify this patch?

@wangyumwangyum mentioned this pull request Nov 10, 2018
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
Closesapache#21766Closesapache#21679Closesapache#21161Closesapache#20846Closesapache#19434Closesapache#18080Closesapache#17648Closesapache#17169
Add:
Closesapache#22813Closesapache#21994Closesapache#22005Closesapache#22463
Add:
Closesapache#15899
Add:
Closesapache#22539Closesapache#21868Closesapache#21514Closesapache#21402Closesapache#21322Closesapache#21257Closesapache#20163Closesapache#19691Closesapache#18697Closesapache#18636Closesapache#17176Closesapache#23001 from wangyum/CloseStalePRs.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jinshen-cn@maropu@AmplabJenkins@jacshen-ebay