[Spark-11522][SQL] input_file_name() returns "" for external tables

[xwu0226]

When computing partition for non-parquet relation, HadoopRDD.compute is used. but it does not set the thread local variable inputFileName in NewSqlHadoopRDD, like NewSqlHadoopRDD.compute does.. Yet, when getting the inputFileName, NewSqlHadoopRDD.inputFileName is exptected, which is empty now.
Adding the setting inputFileName in HadoopRDD.compute resolves this issue.

[rxin]

Jenkins, test this please.

[SparkQA]

Test build #2014 has finished for PR 9542 at commit 2658f28.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xwu0226]

@rxin I pushed again for the scala style test issue. Will the test build be kicked off automatically or manually? Thanks!

[squito]

Jenkins, ok to test

[SparkQA]

Test build #45330 has finished for PR 9542 at commit b5fa291.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xwu0226]

@rxin or @squito , what do you think about the fix? Thanks!

[yhuai]

Can you call SqlNewHadoopRDD.unsetInputFileName() in https://github.com/apache/spark/pull/9542/files#diff-83eb37f7b0ebed3c14ccb7bff0d577c2R257? Like what we do in SqlNewHadoopRDD?

[xwu0226]

@yhuai Thanks for pointing it out! I will make the change now.

[SparkQA]

Test build #45911 has finished for PR 9542 at commit c27d030.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Seems we do not need to say where it points to since it is a managed table.

[yhuai]

@xwu0226 Looks good! I left a few comments regarding the format.

[SparkQA]

Test build #45956 has finished for PR 9542 at commit fe2d6d8.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xwu0226]

Accidentially pushed another JIRA's code together. . I am backing it out

[yhuai]

LGTM pending jenkins.

[yhuai]

@xwu0226 Sorry for asking you to update several times. I just realized that you added a bunch of files in sql/hive/src/test/resources/data/. Since that file is directly copied from hive, we do not change files or add files in there. Can we just generate some test files in the test? We can make HiveUDFSuite extend SQLTestUtils and then use withTempPath to generate temp dirs that can be used for those external tables.

[SparkQA]

Test build #45959 has finished for PR 9542 at commit 4481c82.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xwu0226]

@yhuai I did not know that we should not update the resources/data directory.. I thought the test data files were added along the way by contributors. Thanks for pointing it out! Let me update HiveUDFSuite then.

[yhuai]

@xwu0226 Thank you!

[yhuai]

oh seems there is a conflict...

[xwu0226]

@yhuai Is it mergable?

[yhuai]

@xwu0226 Can you resolve the conflict? Once you update the pr and jenkins is good, I will merge it. Thanks!

[SparkQA]

Test build #45970 has finished for PR 9542 at commit 83b1c77.

• This patch fails PySpark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[yhuai]

test this please

[SparkQA]

Test build #45979 has finished for PR 9542 at commit eeaa6b6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45977 has finished for PR 9542 at commit eeaa6b6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xwu0226]

@yhuai The last test build passed. Do you know what might cause the previous errors? After resolving the conflicts, my own diff for this PR is still the same place, that passed test before. Hope it did not break anything. Thanks!

[yhuai]

@xwu0226https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45977/consoleFull is good. I will merge it to master and branch 1.6.

[xwu0226]

@yhuai Many thanks!