[Python] Failing test with pandas master for extension type conversion

[asfimport]

The pandas master test build has one failure

_______________ test_conversion_extensiontype_to_extensionarray ________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fcd6c580bd0>

 def test_conversion_extensiontype_to_extensionarray(monkeypatch):
 # converting extension type to linked pandas ExtensionDtype/Array
 import pandas.core.internals as _int
 
 storage = pa.array([1, 2, 3, 4], pa.int64())
 arr = pa.ExtensionArray.from_storage(MyCustomIntegerType(), storage)
 table = pa.table({'a': arr})
 
 if LooseVersion(pd.__version__) < "0.26.0.dev":
 # ensure pandas Int64Dtype has the protocol method (for older pandas)
 monkeypatch.setattr(
 pd.Int64Dtype, '__from_arrow__', _Int64Dtype__from_arrow__,
 raising=False)
 
 # extension type points to Int64Dtype, which knows how to create a
 # pandas ExtensionArray
> result = table.to_pandas()

opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/tests/test_pandas.py:3560: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/ipc.pxi:559: in pyarrow.lib.read_message
 ???
pyarrow/table.pxi:1369: in pyarrow.lib.Table._to_pandas
 ???
opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/pandas_compat.py:764: in table_to_blockmanager
 blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/pandas_compat.py:1102: in _table_to_blocks
 for item in result]
opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/pandas_compat.py:1102: in <listcomp>
 for item in result]
opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/pandas_compat.py:723: in _reconstruct_block
 pd_ext_arr = pandas_dtype.__from_arrow__(arr)
opt/conda/envs/arrow/lib/python3.7/site-packages/pandas/core/arrays/integer.py:108: in __from_arrow__
 array = array.cast(pyarrow_type)
pyarrow/table.pxi:240: in pyarrow.lib.ChunkedArray.cast
 ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

> ???
E pyarrow.lib.ArrowNotImplementedError: No cast implemented from extension<arrow.py_extension_type> to int64

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [C++][Python] Support casting an Extension type to its storage type (is blocked by)
• [CI][Crossbow] Pandas integration test fails (duplicates)
• [Python] Builds on master broken by pandas 1.0.2 release (is related to)

PRs and other links:

• GitHub Pull Request #6664

_{Note: This issue was originally created as ARROW-7857. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The test can easily be fixed on our side, but it would also be solved by implementing casting for extension types: ARROW-7858

[asfimport]

Kouhei Sutou / @kou:
Issue resolved by pull request 6664
#6664