[SPARK-16542][SQL][PYSPARK] Fix bugs about types that result an array of null when creating dataframe using python

[zasdfgbnm]

What changes were proposed in this pull request?

Fix bugs about types that result an array of null when creating DataFrame using python.

Python's array.array have richer type than python itself, e.g. we can have array('f',[1,2,3]) and array('d',[1,2,3]). Codes in spark-sql and pyspark didn't take this into consideration which might cause a problem that you get an array of null values when you have array('f') in your rows.

A simple code to reproduce this bug is:

from pyspark import SparkContext
from pyspark.sql import SQLContext,Row,DataFrame
from array import array
sc = SparkContext()
sqlContext = SQLContext(sc)
row1 = Row(floatarray=array('f',[1,2,3]), doublearray=array('d',[1,2,3]))
rows = sc.parallelize([ row1 ])
df = sqlContext.createDataFrame(rows)
df.show()

which have output

+---------------+------------------+
| doublearray| floatarray|
+---------------+------------------+
|[1.0, 2.0, 3.0]|[null, null, null]|
+---------------+------------------+

How was this patch tested?

tested manually

[holdenk]

Oh interesting - thanks for working on this @zasdfgbnm and sorry its sort of fallen through the cracks. Is this something you are still working on? For PRs to get in you generally need some form of automated tests, let me know if you would like some help adding tests for this issue.

[zasdfgbnm]

I'd love to help @holdenk

[zasdfgbnm]

Something to mention is, there is still one problem that I'm not sure whether I solve it correctly: in python's array, unsigned types are supported, but unsigned types are not supported in JVM. The solution in this PR is to convert unsigned types to a larger type, e.g. unsigned int -> long. I'm not sure whether it would be better to reject the unsigned types in python and throw an exception.

[zasdfgbnm]

Hi @holdenk , I think I'm done. I create a test for this issue and I do find from the test that spark has the same issue not only for float but also for byte and short. After several commits, ./python/run-tests --modules=pyspark-sql passes on my computer.

To be clear, I need to say that only array with typecode b,h,i,l,f,d are supported, array with typecode u is not supported because it "corresponds to Python’s obsolete unicode character", array with typecode B,H,I,L are not supported because there is no unsigned types on JVM, array with typecode q,Q are not supported because they "are available only if the platform C compiler used to build Python supports C long long", which makes supporting them complicated. For the unsupported typecodes, a TypeError will be raised if the user try to create a DataFrame of it.

Would you, or any other developer, review my code and get it merged?

[gatorsmile]

cc @ueshin

[ueshin]

@holdenk Are you still working on this? If so, could you rebase or merge master to fix conflicts please?
Oops, I'm sorry, I made a mistake to ping to.

[ueshin]

@zasdfgbnm Are you still working on this? If so, could you rebase or merge master to fix conflicts please?

[gatorsmile]

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

[zasdfgbnm]

@ueshin@gatorsmile I'm happy to resolve the conflicts IF AND ONLY IF there will be a developer work on the code review for this. This PR was opened more than a year ago and I keep waiting for the review for one year. If it is guaranteed that there will be a reviewer assigned for this recently, I will resolve the conflicts. Otherwise, I don't want to maintain a PR forever just to wait for review.

[felixcheung]

@zasdfgbnm I think you can ping @ueshin to review.
Sounds important to me to have. Ping me if it falls through.

[gatorsmile]

@zasdfgbnm Please reopen the PR and @ueshin can help review your PR. Thanks!

[zasdfgbnm]

reopened at #18444

[gatorsmile]

retest this please