If you create a Table from a DataFrame of ints with a NaN value the NaN is improperly cast. Since pandas casts these to floats, when converted to a table the NaN is interpreted as an integer. This seems like a bug since a known limitation in pandas (the inability to have null valued integers data) is taking precedence over arrow's functionality to store these as an IntArray with nulls.
importpyarrowaspaimportpandasaspddf = pd.DataFrame({"a":[1, 2, pd.np.NaN]})
schema = pa.schema([pa.field("a", pa.int64(), nullable=True)])
table = pa.Table.from_pandas(df, schema=schema)
table[0]
<pyarrow.lib.Columnobjectat0x7f2151d19c90>
chunk0: <pyarrow.lib.Int64Arrayobjectat0x7f213bf356d8>
[
1,
2,
-9223372036854775808
]Reporter: Matthew Gilbert
Assignee: Antoine Pitrou / @pitrou
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-2135. Please see the migration documentation for further details.
If you create a
Tablefrom aDataFrameof ints with a NaN value the NaN is improperly cast. Since pandas casts these to floats, when converted to a table the NaN is interpreted as an integer. This seems like a bug since a known limitation in pandas (the inability to have null valued integers data) is taking precedence over arrow's functionality to store these as an IntArray with nulls.Reporter: Matthew Gilbert
Assignee: Antoine Pitrou / @pitrou
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-2135. Please see the migration documentation for further details.