[SPARK-4985][SQL Parquet] Parquet date support

[alexbaretta]

No description provided.

[ash211]

Jenkins, this is ok to test

[SparkQA]

Test build #24979 has finished for PR 3855 at commit d29224e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nchammas]

cc @marmbrus

[SparkQA]

Test build #25509 has finished for PR 3855 at commit 8851d1a.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #25510 has finished for PR 3855 at commit 929f294.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Nit: Usually I'd prefer import java.sql.Date and just use Date here.

[marmbrus]

Seems like this is subsumed by #3822, and thus we can close this issue.

[rdblue]

This doesn't conform to the Parquet specification for date and produces invalid data.

When using the DATE annotation, the value must be the number of days from the Unix epoch, 1 January 1970. java.sql.Date and java.util.Date are backed by a long timestamp, the number of milliseconds from the Unix epoch (which is a Parquet TIMESTAMP_MILLIS) and casting that value to an integer makes it impossible to recover the real date.

I recommend using TIMESTAMP_MILLIS instead of date here (you won't need the toInt part). That seems to be what you want, if you're interested in using java.sql.Date. The reason why there is a name mismatch is that the Parquet types mirror SQL types more closely than Java objects.

[rdblue]

I just looked at #3822 and it looks correct, so you can ignore my review comment above. In the future, please feel free to ping me for reviews when you're using logical types like Date, Timestamp, Decimal, etc. in either Parquet or Avro. I've been working on the specs in those communities and I'm happy to make sure the implementations look correct.

[liancheng]

@rdblue Thanks all the same for the review! BTW, it would be great if you can have a look at #5422, which refactors Spark SQL Parquet converter (Parquet records to Spark SQL rows) and implements backwards-compatibility rules.

[rdblue]

@liancheng, I'll take a look as soon as I can. I'm a little swamped this week though, so I can't guarantee it'll be quick. Sorry!