[SPARK-18134][SQL] Comparable MapTypes [POC]

[hvanhovell]

What changes were proposed in this pull request?

This is a small POC to see if we can make MapType orderable, and thus usable in aggregates and joins. The key idea in this PR is that there is a difference between an unordered and an ordered map (an ordered map is can be compared), and that ordered is a property of MapType.

A map can be converted from an unordered map to an ordered map by injecting a SortMap expression. The analyzer will inject SortMap expressions whenever we use a map in a binary comparison and when we use it in an aggregate. Note that the SortMap expression is far from optimized, it should however perform reasonable.

How was this patch tested?

No tests yet. This will probably fail tests.

[hvanhovell]

cc @cloud-fan we discussed something like this earlier today

[SparkQA]

Test build #68968 has finished for PR 15970 at commit edec2d8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class SortMap(child: Expression) extends UnaryExpression with ExpectsInputTypes

[SparkQA]

Test build #69003 has finished for PR 15970 at commit b16ddb8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #69023 has finished for PR 15970 at commit d570c84.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #69136 has finished for PR 15970 at commit e4cc4b0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class ColumnStat(
• case class UncacheTableCommand(

[SparkQA]

Test build #69141 has finished for PR 15970 at commit 7d847b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[albertferras]

Any update on this?

[gatorsmile]

cc @hvanhovell

[janewangfb]

Is there any updates on this PR? This feature is quite useful to us.

[jinxing64]

@hvanhovell Are you still working on this? I think this is feature is useful :)

[maropu]

@hvanhovell We still need to keep this pr open? Either way, we need rework based on this pr. If so, can you close this for now?

[SparkQA]

Test build #97829 has started for PR 15970 at commit 7d847b8.

[SparkQA]

Test build #97847 has started for PR 15970 at commit 7d847b8.

[SparkQA]

Test build #97859 has started for PR 15970 at commit 7d847b8.

[AmplabJenkins]

Build finished. Test FAILed.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!