[Doc] [SQL] Addes Hive metastore Parquet table conversion section

[liancheng]

This PR adds a section about Hive metastore Parquet table conversion. It documents:

1. Schema reconciliation rules introduced in [SPARK-6538][SQL] Add missing nullable Metastore fields when merging a Parquet schema #5214 (see this comment in [SPARK-6538][SQL] Add missing nullable Metastore fields when merging a Parquet schema #5188)
2. Metadata refreshing requirement introduced in [SPARK-6575][SQL] Converted Parquet Metastore tables no longer cache metadata #5339

[SparkQA]

Test build #29669 has started for PR 5348 at commit 9840aff.

[liancheng]

Removed the backquotes because the rendered text looks funny.

[liancheng]

Added Python snippet for metadata refreshing after adding refreshTable for PySpark in #5349.

[SparkQA]

Test build #29670 has started for PR 5348 at commit a56483a.

[SparkQA]

Test build #29669 has finished for PR 5348 at commit 9840aff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29669/
Test PASSed.

[yhuai]

I just found we have an example having the old code (https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#programmatically-specifying-the-schema). We still have the following

// Import Spark SQL data types and Row.
import org.apache.spark.sql._

Can you update it to use import org.apache.spark.sql.types._ for data types?

[SparkQA]

Test build #29670 has finished for PR 5348 at commit a56483a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29670/
Test PASSed.

[liancheng]

@yhuai This typo has been fixed in master and branch-1.3.

[SparkQA]

Test build #29674 has started for PR 5348 at commit 22d7b14.

[yhuai]

Seems it is under Hive metastore Parquet table conversion. However, users may need to call refresh table in other cases, right? For example, when they manually copy data to the dir of a data source table.

[SparkQA]

Test build #29674 has finished for PR 5348 at commit 22d7b14.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29674/
Test PASSed.

[SparkQA]

Test build #31029 has started for PR 5348 at commit 22d7b14.

[andrewor14]

@liancheng@yhuai what happened here? Do we still want to merge this?

[liancheng]

@yhuai I guess it's OK to merge this?

[yhuai]

@liancheng Yeah, it's fine to merge it. btw, you have added refreshTable in Python, right? We can update the description. Also, can you build the doc to double check the generated doc looks fine?

[liancheng]

@yhuai Finished updating Python refreshTable snippet and double checking rendered docs. Will merge this pending Jenkins.

[AmplabJenkins]

Merged build triggered.

[yhuai]

Sure. Thanks!

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #35586 has started for PR 5348 at commit 42ae0d0.

[andrewor14]

(you could just merge this since Jenkins doesn't actually test the docs)

[SparkQA]

Test build #35586 has finished for PR 5348 at commit 42ae0d0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test FAILed.

[liancheng]

@andrewor14 Yeah... Thanks :) Merging to master.