[Python] Add safe option to Table.from_pandas to avoid unsafe casts

[asfimport]

Ported over from #2217

In [8]: import pandas as pd
 ...: import pyarrow as arw

In [9]: df = pd.DataFrame({'A': list('abc'), 'B': np.arange(3)})
 ...: df
Out[9]:
 A B
0 a 0
1 b 1
2 c 2

In [10]: schema = arw.schema([
 ...: arw.field('A', arw.string()),
 ...: arw.field('B', arw.int32()),
 ...: ])

In [11]: tbl = arw.Table.from_pandas(df, preserve_index=False, schema=schema)
 ...: tbl
Out[11]:
pyarrow.Table
A: string
B: int32
metadata
--------
{b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [{"name":'
 b' "A", "field_name": "A", "pandas_type": "unicode", "numpy_type":'
 b' "object", "metadata": null}, {"name": "B", "field_name": "B", "'
 b'pandas_type": "int32", "numpy_type": "int32", "metadata": null}]'
 b', "pandas_version": "0.23.1"}'}

In [12]: tbl.to_pandas().equals(df)
Out[12]: True

...so if the schema matches the pandas datatypes all is well - we can roundtrip the DataFrame.

Now, say we have some bad data such that column 'B' is now of type float64. The datatypes of the DataFrame don't match the explicitly supplied schema object but rather than raising a TypeError the data is silently truncated and the roundtrip DataFrame doesn't match our input DataFame without even a warning raised!

In [13]: df['B'].iloc[0] = 1.23
 ...: df
Out[13]:
 A B
0 a 1.23
1 b 1.00
2 c 2.00

In [14]: # I would expect/want this to raise a TypeError since the schema doesn't match the pandas datatypes
 ...: tbl = arw.Table.from_pandas(df, preserve_index=False, schema=schema)
 ...: tbl
Out[14]:
pyarrow.Table
A: string
B: int32
metadata
--------
{b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [{"name":'
 b' "A", "field_name": "A", "pandas_type": "unicode", "numpy_type":'
 b' "object", "metadata": null}, {"name": "B", "field_name": "B", "'
 b'pandas_type": "int32", "numpy_type": "float64", "metadata": null'
 b'}], "pandas_version": "0.23.1"}'}

In [15]: tbl.to_pandas() # <-- SILENT TRUNCATION!!!
Out[15]:
 A B
0 a 1
1 b 1
2 c 2

To be clear, I would really like Table.from_pandas to raise a TypeError if the DataFrame types don't match an explicitly supplied schema and would hope this current behaviour would be considered a bug.

Reporter: Dave Hirschfeld / @dhirschfeld
Assignee: Krisztian Szucs / @kszucs

Related issues:

• [Python/C++] Add option to Array.from_pandas and pyarrow.array to perform unsafe casts (depends upon)
• [C++] Handle float truncation during casting (depends upon)

PRs and other links:

• GitHub Pull Request #2504

_{Note: This issue was originally created as ARROW-2799. Please see the migration documentation for further details.}

[asfimport]

Dave Hirschfeld / @dhirschfeld:
The recommendation by wesm was to add a safe/unsafe option to Table.from_pandas

[asfimport]

Wes McKinney / @wesm:
I renamed the issue slightly

[asfimport]

Krisztian Szucs / @kszucs:
NOTE: We need a castoption for the float case.

[asfimport]

Wes McKinney / @wesm:
Maybe sooner than later we should expose more granular cast options in all of these functions

[asfimport]

Uwe Korn / @xhochy:
Issue resolved by pull request 2504
#2504