[SPARK-3138][SQL] sqlContext.parquetFile should be able to take a single file as parameter

[chutium]

if (!fs.getFileStatus(path).isDir) throw Exception make no sense after this commit #1370

be careful if someone is working on SPARK-2551, make sure the new change passes test case test("Read a parquet file instead of a directory")

[SparkQA]

QA tests have started for PR 2044 at commit 4ae477f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2044 at commit 4ae477f.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

A normal, complete Parquet file on HDFS should be directory with a _metadata file. If I understand this PR correctly, it actually tries to handle a special yet abnormal case. Personally, I don't think we should encourage users to generate and/or use potentially broken file segments.

[chutium]

i am confused about this definition "Parquet file on HDFS should be directory"...

this was reported in user list http://apache-spark-user-list.1001560.n3.nabble.com/sqlContext-parquetFile-path-fails-if-path-is-a-file-but-succeeds-if-a-directory-td12345.html

i think he is also right, parameter of sqlContext.parquetFile should be a dir of parquet files (better with a _metadata file in it), but this should not be a mandatory requirement, our javadoc
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SQLContext.html#parquetFile(java.lang.String)
says also, this method "Loads a Parquet file", not "loads a dir of parqauet files"...

otherwise maybe we should rename this method to sqlContext.parquetDir or somthing

and this PR is not "try to handle", it is more like "try to ignore useless check" :) it is already handled by children.find(...)

[marmbrus]

I tend to agree here with attempting to provide more functionality as long as it doesn't complicate the code base. Seems like parquet works fine without this check.

[marmbrus]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 2044 at commit 4ae477f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2044 at commit 4ae477f.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command

[liancheng]

Hmm... the last Jenkins build info has already been cleaned up, don't know why it failed.

[liancheng]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2044 at commit 4ae477f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2044 at commit 4ae477f.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

The build failure was caused by unrelated test suites, should be good to go.

[marmbrus]

Thanks! I've merged this into master and 1.1.

[mohnishkodnani]

Can we have this take a parent directory that has other subdirectories with parquet files, instead of the parquet files being the immediate children of the directory.