Uh oh!
There was an error while loading. Please reload this page.
[SPARK-5498][SQL][FOLLOW] add schema to table partition - #20846
[SPARK-5498][SQL][FOLLOW] add schema to table partition#20846liutang123 wants to merge 1 commit into
Conversation
| parameters: Map[String, String] = Map.empty, | ||
| stats: Option[CatalogStatistics] = None) { | ||
| stats: Option[CatalogStatistics] = None, | ||
| schema: Option[StructType] = None) { |
There was a problem hiding this comment.
The partition schema is stored in CatalogTable . I am not very clear what is the exception you got.
@dongjoon-hyun Could you help @liutang123 investigate the issue?
There was a problem hiding this comment.
Sure, @gatorsmile . I'll take a look during weekend.
There was a problem hiding this comment.
Some times, partition's schema is different from the table's.
dongjoon-hyun
commented
Mar 17, 2018
@liutang123 , Spark should not do this kind of risky thing. Hive 2.3.2 also disallows incompatible schema changes like the following. hive> CREATE TABLE test_par(a string) PARTITIONED BY (b bigint) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';
OK
Time taken: 0.262 seconds
hive> ALTER TABLE test_par CHANGE a a bigint RESTRICT;
FAILED: Execution Error, return code 1fromorg.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions :
a
hive>SELECT VERSION();
OK
2.3.2 r857a9fd8ad725a53bd95c1b2d6612f9b1155f44d
Time taken: 0.711 seconds, Fetched: 1 row(s)cc @gatorsmile . |
@dongjoon-hyun, thanks for reviewing. |
gatorsmile
commented
Mar 19, 2018
We do not allow users to change the table column type. Currently, only the column comments are allowed to change if users issue the command through Spark. However, users still can change it through Hive. Thus, nothing we can do from Spark side, right? |
dongjoon-hyun
commented
Mar 19, 2018
Right, @gatorsmile . |
AmplabJenkins
commented
Oct 22, 2018
Can one of the admins verify this patch? |
srowen
commented
Nov 10, 2018
What JIRA was this really about? |
Closesapache#21766Closesapache#21679Closesapache#21161Closesapache#20846Closesapache#19434Closesapache#18080Closesapache#17648Closesapache#17169 Add: Closesapache#22813Closesapache#21994Closesapache#22005Closesapache#22463 Add: Closesapache#15899 Add: Closesapache#22539Closesapache#21868Closesapache#21514Closesapache#21402Closesapache#21322Closesapache#21257Closesapache#20163Closesapache#19691Closesapache#18697Closesapache#18636Closesapache#17176Closesapache#23001 from wangyum/CloseStalePRs. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>
What changes were proposed in this pull request?
When query a orc table witch some partition schemas are different from table schema, ClassCastException will occured.
reproduction:
create table test_par(a string) PARTITIONED BY (bbigint) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';ALTER TABLE test_par CHANGE a a bigint restrict; -- in hiveselect * from test_par;How was this patch tested?
manual test.