[SPARK-25659][PYTHON][TEST] Test type inference specification for createDataFrame in PySpark

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to specify type inference and simple e2e tests. Looks we are not cleanly testing those logics.

For instance, see

spark/python/pyspark/sql/types.py

Lines 894 to 905 in 08c76b5

	_type_mappings= {
	type(None): NullType,
	bool: BooleanType,
	int: LongType,
	float: DoubleType,
	str: StringType,
	bytearray: BinaryType,
	decimal.Decimal: DecimalType,
	datetime.date: DateType,
	datetime.datetime: TimestampType,
	datetime.time: TimestampType,
	}

Looks we intended to support datetime.time and None for type inference too but it does not work:

>>> spark.createDataFrame([[datetime.time()]])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/sql/session.py", line 751, in createDataFrame
rdd, schema = self._createFromLocal(map(prepare, data), schema)
File "/.../spark/python/pyspark/sql/session.py", line 432, in _createFromLocal
data = [schema.toInternal(row) for row in data]
File "/.../spark/python/pyspark/sql/types.py", line 604, in toInternal
for f, v, c in zip(self.fields, obj, self._needConversion))
File "/.../spark/python/pyspark/sql/types.py", line 604, in <genexpr>
for f, v, c in zip(self.fields, obj, self._needConversion))
File "/.../spark/python/pyspark/sql/types.py", line 442, in toInternal
return self.dataType.toInternal(obj)
File "/.../spark/python/pyspark/sql/types.py", line 193, in toInternal
else time.mktime(dt.timetuple()))
AttributeError: 'datetime.time' object has no attribute 'timetuple'
>>> spark.createDataFrame([[None]])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/sql/session.py", line 751, in createDataFrame
rdd, schema = self._createFromLocal(map(prepare, data), schema)
File "/.../spark/python/pyspark/sql/session.py", line 419, in _createFromLocal
struct = self._inferSchemaFromList(data, names=schema)
File "/.../python/pyspark/sql/session.py", line 353, in _inferSchemaFromList
raise ValueError("Some of types cannot be determined after inferring")
ValueError: Some of types cannot be determined after inferring

How was this patch tested?

Manual tests and unit tests were added.

[SparkQA]

Test build #97042 has finished for PR 22653 at commit 8fddb9d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class A(object):

[HyukjinKwon]

@BryanCutler and @viirya would you mind if I ask to take a look please?

[BryanCutler]

LGTM, would you mind creating JIRAs for fixing inference from None and datetime.time?

[BryanCutler]

I didn't this was possible - does it just look at the variable attributes in the object to get the fields?

[HyukjinKwon]

Yea, it uses __dict__ attribute.. looks that's not possible in UDFs .. possibly an issue.

[HyukjinKwon]

Let me open a JIRA after taking some more further looks. I checked some code places and looks it's not easy to support None for instance.

[HyukjinKwon]

Merged to master.

Thank you @BryanCutler. Let me try to make sure I take further actions for related items.

[viirya]

Sorry for late. @HyukjinKwon Yes, this LGTM. It is great we can follow up other issues like None and datetime.time in other JIRA.

[viirya]

nit: since this is for unicode string, how about using a non-ascii string?

[HyukjinKwon]

Fair point. Will change when I fix some codes around here.