ARROW-7663: [Python] Raise better error message when passing mixed-type (int/string) Pandas dataframe to pyarrow Table

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!

[arw2019]

This PR homogenizes error messages for mixed-type Pandas inputs to pa.Table.

The message for Pandas column with int followed by string is now

In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')

the same as for double followed by string:

In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']})) (... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')

As a side effect, this snippet [xref #5866, ARROW-7168] now throws an ArrowInvalid (has been FutureWarning since 0.16):

In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object)) ...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64()) ...: result = pa.array(cat, type=typ) (... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int

Finally, this does break a test [xref #4484, ARROW-4036] - see code comment

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7663

[arw2019]

We lose the the more specific traceback and ZeroDivisionError message, in favor of

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/0In [12]: pa.array([MyBrokenInt()], type=pa.int64()) ---------------------------------------------------------------------------ArrowInvalidTraceback (mostrecentcalllast)
<ipython-input-12-1cf156b165b3>in<module>---->1pa.array([MyBrokenInt()], type=pa.int64())
~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib.array()
269else:
270# ConvertPySequence does strict conversion if type is explicitly passed-->271return_sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272273~/git_repo/arrow/python/pyarrow/array.pxiinpyarrow.lib._sequence_to_array()
3839withnogil:
--->40check_status(ConvertPySequence(sequence, mask, options, &out))
4142ifout.get().num_chunks() ==1:
~/git_repo/arrow/python/pyarrow/error.pxiinpyarrow.lib.check_status()
8283ifstatus.IsInvalid():
--->84raiseArrowInvalid(message)
85elifstatus.IsIOError():
86# Note: OSError constructor isArrowInvalid: Couldnotconvert<__main__.MyBrokenIntobjectat0x7fc331394290>withtypeMyBrokenInt: triedtoconverttoint

but this is the same message as what we get on master for

In [11]: classMyBrokenInt: ...: def__init__(self): ...: 1/1

so maybe it's ok?

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

What did the error message say before, and what does it show now?

[arw2019]

On master it's

TypeError: anintegerisrequired (gottypepyarrow.lib.Int8Array)

verus on this branch

ArrowInvalid: Couldnotconvert [
5
] withtypepyarrow.lib.Int8Array: triedtoconverttoint

[jorisvandenbossche]

Hmm, for this case I find the original error message clearer ..

That's the consequence of the scalar(..) conversion using the array conversion under the hood, I suppose?
But OK, I suppose this is fine (it's maybe mainly the multiline repr of the array in the middle of the sentence that makes it more confusing)

[jorisvandenbossche]

I think that is fine, personally

[jorisvandenbossche]

What are the cases that this couldn't be converted, but that objis an integer? When the integer is too big to fit in a C int?

[arw2019]

Yes, and also when converting a negative integer to a uint:

pa.scalar(-1, type='uint8')

No other tests are touched if I recompile without this check

[emkornfield]

@jorisvandenbossche was there more to be done here?

[jorisvandenbossche]

Thanks for the ping. I think all good. @arw2019 can you just rebase to ensure it's still all passing with latest master?

[arw2019]

@jorisvandenbossche Rebased and seeing some failures. They're ones also popping up in other, unrelated, PRs, so not sure they're to do with this patch? I'm happy to investigate, though

[jorisvandenbossche]

There are some known failures on Mac and Appveyor at the moment, so nothing to worry about for this PR.

[jorisvandenbossche]

Thanks @arw2019 !

[arw2019]

Thanks @jorisvandenbossche for reviewing!