[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile

[ghost]

fix compile error on hadoop 0.23 for the pull request #1924.

[AmplabJenkins]

Can one of the admins verify this patch?

[marmbrus]

ok to test

[SparkQA]

QA tests have started for PR 1959. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18586/consoleFull

[SparkQA]

QA results for PR 1959:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18586/consoleFull

[srowen]

How about ignoring any file starting with _ ? Hadoop (also) uses this convention, for things like the _SUCCESS file.

[ghost]

Unfortunately, that would ignore the metadata file "_metadata" as well.

[chutium]

maybe we should rethink about why we use filterNot here? simple filter works fine here, something like:

val children = fs.listStatus(path).filter { status =>
  val name = status.getPath.getName
  name == ParquetFileWriter.PARQUET_METADATA_FILE || (name(0) != '.' && name(0) != '_')
}

so we can ignore all of hidden/tmp files without _metadata

[liancheng]

I agree with this. Just remove .* and _* except _metadata.

[SparkQA]

QA tests have started for PR 1959 at commit be30793.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1959 at commit be30793.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

LGTM, thanks.

[marmbrus]

Thanks! I've merged this into master and 1.1.