[SPARK-11692][SQL] Support for Parquet logical types, JSON and BSON (embedded types)

[HyukjinKwon]

Parquet supports some JSON and BSON datatypes. They are represented as binary for BSON and string (UTF-8) for JSON internally.

I searched a bit and found Apache drill also supports both in this way, link.

[HyukjinKwon]

retest this please

[HyukjinKwon]

cc @liancheng

[SparkQA]

Test build #45724 has finished for PR 9658 at commit d5a9629.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45726 has finished for PR 9658 at commit d5a9629.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Hm, again, what's the +?

[SparkQA]

Test build #45732 has finished for PR 9658 at commit 9f22651.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45731 has finished for PR 9658 at commit 66088e3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45804 has finished for PR 9658 at commit d17a2db.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #45827 has finished for PR 9658 at commit d17a2db.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #45835 has finished for PR 9658 at commit d17a2db.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

All the builds pass all the tests at ParquetIOSuite and I do not think it affects other modules such as ML.
I will retest this.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #45843 has finished for PR 9658 at commit d17a2db.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Sorry that I missed this part during the last review. Please always use === instead of == for better assertion error messages.

[HyukjinKwon]

Thanks! I changed this.

[SparkQA]

Test build #45963 has finished for PR 9658 at commit 1152636.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Thanks! Merging to master.

[marmbrus]

We've added support for reading these types as strings, but we can't round trip data without losing the annotation which might be kind of confusing for users. Perhaps we should also be reading/writing this info to/from the metadata.

[HyukjinKwon]

Please note that #9754 updated unintentionally this to clean up at mater branch however, that is supposed to be merged with branch 1.6 and for this version 1.7.
I will give an alert to you or make a PR to backport this as soon as branch 1.7 is available.

[liancheng]

Hm, I don't quite get it... So this PR is only for master (targeting 1.7). I don't think we need to backport this one to anywhere else.

[HyukjinKwon]

Ah. I just got confused for a bit. It doesn't need to.

[liancheng]

@marmbrus Did you mean the metadata stored in Parquet key-value user defined metadata, or the schema metadata in StructField? For primitive JSON/BSON fields, it's a good idea to annotate them in the schema metadata. But it doesn't cover nested fields, e.g. an array field of JSON/BSON elements.

And for "support for reading these types as strings", are you referring to spark.sql.parquet.binaryAsString? Currently this option only converts Parquet binaries without any logical type annotations, thus it doesn't cover binary (JSON) or binary (BSON).

[marmbrus]

But it doesn't cover nested fields, e.g. an array field of JSON/BSON elements.

That is an unfortunate limitation of our metadata, but it does seem like it could be worked around. Though that said this is a minor concern.

"support for reading these types as strings"

I'm just saying thats what this patch does. It just reads them in as a text/binary string of opaque bytes.