[SPARK-16397][SQL] make CatalogTable more general and less hive specific

[cloud-fan]

What changes were proposed in this pull request?

CatalogTable is an internal interface to represent table metadata in Spark SQL, and should be able to express both hive serde table and data source table. Due to some hive hacks, data source table only use table properties to store its information, and the CatalogStorageFormat is very hive specific as it's only used by hive serde table currently.

However this is not a good design, the CatalogTable is not general enough to express data source table if we remove hive hacks in the future.

This PR is trying to make CatalogStorageFormat general enough to handle all types of tables in Spark SQL.

How was this patch tested?

existing tests.

[SparkQA]

Test build #61858 has finished for PR 14071 at commit 4ef7dd0.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

storageProps += "inputFormat" -> string(c.inFmt)? string(c.inFmt) throws an NPE if it is null; so option is redundant...

[cloud-fan]

cc @yhuai@gatorsmile@liancheng@clockfly

[SparkQA]

Test build #61914 has finished for PR 14071 at commit 4d65609.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Any reason why this locationUri is set to None? It sounds like the original value is Some(relation.location.paths.map(_.toUri.toString).head

[cloud-fan]

oh it's a mistake...

[SparkQA]

Test build #61951 has finished for PR 14071 at commit 9efe46e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61959 has finished for PR 14071 at commit b218bb7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

We never persist this flag into external catalog.

[SparkQA]

Test build #62247 has finished for PR 14071 at commit 632d9a7.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #62249 has finished for PR 14071 at commit d18ac59.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I'm closing it as it turns out that we still need these hive specific stuff in CatalogStorageFormat. Hiding them in properties seems don't have much benefit, what we need is just adding a 'provider' field in CatalogTable, which is done in #14155