[SPARK-26355][PYSPARK] Add a workaround for PyArrow 0.11.

[ueshin]

What changes were proposed in this pull request?

In PyArrow 0.11, there is a API breaking change.

• ARROW-1949 - [Python/C++] Add option to Array.from_pandas and pyarrow.array to perform unsafe casts.

This causes test failures in ScalarPandasUDFTests.test_vectorized_udf_null_(byte|short|int|long):

 File "/Users/ueshin/workspace/apache-spark/spark/python/pyspark/worker.py", line 377, in main
process()
File "/Users/ueshin/workspace/apache-spark/spark/python/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/Users/ueshin/workspace/apache-spark/spark/python/pyspark/serializers.py", line 317, in dump_stream
batch = _create_batch(series, self._timezone)
File "/Users/ueshin/workspace/apache-spark/spark/python/pyspark/serializers.py", line 286, in _create_batch
arrs = [create_array(s, t) for s, t in series]
File "/Users/ueshin/workspace/apache-spark/spark/python/pyspark/serializers.py", line 284, in create_array
return pa.Array.from_pandas(s, mask=mask, type=t)
File "pyarrow/array.pxi", line 474, in pyarrow.lib.Array.from_pandas
return array(obj, mask=mask, type=type, safe=safe, from_pandas=True,
File "pyarrow/array.pxi", line 169, in pyarrow.lib.array
return _ndarray_to_array(values, mask, type, from_pandas, safe,
File "pyarrow/array.pxi", line 69, in pyarrow.lib._ndarray_to_array
check_status(NdarrayToArrow(pool, values, mask, from_pandas,
File "pyarrow/error.pxi", line 81, in pyarrow.lib.check_status
raise ArrowInvalid(message)
ArrowInvalid: Floating point value truncated

We should add a workaround to support PyArrow 0.11.

How was this patch tested?

In my local environment.

[ueshin]

cc @BryanCutler@HyukjinKwon

[SparkQA]

Test build #100068 has finished for PR 23305 at commit bb54b74.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

LGTM, I tested and just followed the whole context via linked JIRA.

[HyukjinKwon]

Merged to master and branch-2.4.

[BryanCutler]

late +1, thanks @ueshin !