[SPARK-6055] [PySpark] fix incorrect __eq__ of DataType

[davies]

The eq of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.

Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).

This PR also improve the performance of inferSchema (avoid the unnecessary converter of object).

cc @pwendell@JoshRosen

[davies]

This PR works for 1.3+, will create another PR for 1.2 and 1.1

[SparkQA]

Test build #28053 has started for PR 4808 at commit d9ae973.

• This patch merges cleanly.

[davies]

These are duplicated, also in types.py.

[JoshRosen]

Yep, good catch.

[SparkQA]

Test build #28053 has finished for PR 4808 at commit d9ae973.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28053/
Test FAILed.

[SparkQA]

Test build #28072 has started for PR 4808 at commit 46999dc.

• This patch merges cleanly.

[SparkQA]

Test build #28072 has finished for PR 4808 at commit 46999dc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28072/
Test PASSed.

[SparkQA]

Test build #28079 has started for PR 4808 at commit 534ac90.

• This patch merges cleanly.

[SparkQA]

Test build #28079 has finished for PR 4808 at commit 534ac90.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28079/
Test FAILed.

[SparkQA]

Test build #28084 has started for PR 4808 at commit 3da44fc.

• This patch merges cleanly.

[SparkQA]

Test build #28084 has finished for PR 4808 at commit 3da44fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28084/
Test FAILed.

[JoshRosen]

Is this a breaking API change? Or were the old doctests showing incorrect usage of the API?

[davies]

Old tests are incorrect.

[JoshRosen]

It looks like _restore_object still tries to use DataType instance ids as _cached_cls dictionary keys during unpickling; is this still necessary if the DataTypes aren't singletons after unpickling?

[davies]

@JoshRosen Because we serialized the objects in batch, and pickle memorize the multiple occurrences of same object in the batch, finally we will get single DataType object (even for StructType), we can benefits from this optimization, no __hash__ and __eq__ for later row in the batch.

[SparkQA]

Test build #28094 has started for PR 4808 at commit 6a322a4.

• This patch merges cleanly.

[JoshRosen]

Makes sense; LGTM. I'll take a look at the backport patches, too.

[SparkQA]

Test build #28094 has finished for PR 4808 at commit 6a322a4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28094/
Test PASSed.

[JoshRosen]

LGTM, so I've merged this into branch-1.3 (1.3.0) and master (1.4.0). Thanks!