[SPARK-7626][SQL] Consider the columns in Hive table but not in partition when filling values

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-7626

[SparkQA]

Test build #32707 has finished for PR 6146 at commit 4dec469.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Please explain your change. Neither the JIRA nor the PR description describe why you the original code is incorrect.

[viirya]

When a non-partition key attribution doesn't exist in the partition's (determined by the partition's StructObjectInspector), we should produce a null ref. Previously, we don't check it and directly call getStructFieldRef to query the field ref. It will cause the error reported in the JIRA.

[SparkQA]

Test build #35281 has finished for PR 6146 at commit 21e3c2c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35292 has finished for PR 6146 at commit eed7c8b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SerializableConfiguration(@transient var value: Configuration) extends Serializable
  • class SerializableJobConf(@transient var value: JobConf) extends Serializable

[marmbrus]

What about a test case?

[viirya]

@marmbrus Although the code looks problematic, I mentioned on the JIRA that I can't reproduce the problem. By more testing and searching codes, I found that in your PR #5876 (https://github.com/apache/spark/pull/5876/files#diff-ee66e11b56c21364760a5ed2b783f863R620) you modified how to produce hive partition objects where you populate partition's schema from table schema.

So any newly added columns in hive table will appear in its partitions' schema too. That is why I can't produce this problem. Thus, all non-partition key attributions will always exist in table partitions. And this reported bug is solved by the PR #5876. Because of that, I think I can close this PR now.

[marmbrus]

Thanks for following up!