ARROW-2135: [Python] Fix NaN conversion when casting from Numpy array

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions

[pitrou]

No description provided.

[pitrou]

For some reason (macro expansion?) these #ifs wouldn't work correctly here, even though NPY_INT64 is defined to NPY_LONG.

[pitrou]

Hmm, actually, that must be because NPY_LONGLONG is not a macro...

[cpcloud]

std::fill(null_bitmap_data_, null_bitmap_data_ + null_bytes, 0) is a bit more idiomatic.

[pitrou]

Hmm, perhaps. This is really a copy/paste of NumPyConverter::InitNullBitmap()...

[cpcloud]

Possibly time for a subclass then?

[cpcloud]

Is there already a test for things like a = [1.0, 2.0, 3.1, np.nan] where a user passes in an integer type?

[pitrou]

You mean for the truncation behavior? Let me look.

[pitrou]

No, I don't think so. I'm not sure we specify the truncation mode anywhere either?

[cpcloud]

It looks like it's a hard cast:

In [7]: pa.array([1, 2, 3.190, np.nan], type=pa.int64())
Out[6]:
<pyarrow.lib.Int64Array object at 0x7f537e42dd68>
[
1,
2,
3,
NA
]

[cpcloud]

That's fine. Was just wondering.

[cpcloud]

inline is redundant here: http://en.cppreference.com/w/cpp/language/inline.

A function defined entirely inside a class/struct/union definition, whether it's a member function or a non-member friend function, is implicitly an inline function.

[pitrou]

I see. This is really using the same convention as the rest of the file, though.

[cpcloud]

Hm, so that's also called isnull. Shouldn't that mean v == Py_None?

[cpcloud]

Probably needs a test as well since it isn't failing.

[pitrou]

Nice catch :-) I'm not sure how to test it. Defining isnull is necessary for compiling, but that path isn't taken at runtime as object arrays are handled separately.

[cpcloud]

At some point we may want to have an STL-compatible view class that makes interacting with iterators constructs in the STL much easier. We have a lot of code that is manually handling iteration using a size/count and a buffer.

[pitrou]

Which iterators are you thinking about? Do you mean the ndarray 1d iterator?

[cpcloud]

That's one, though I added begin()/end() for that in #1651.

[pitrou]

By the way, I don't know what that is, but this is required to have the tests pass. Why do we always treat NaT as null but not floating-point NaN? @wesm

[cpcloud]

AFAIU There's no other way to interpret NaT other than NULL (unless there's a standard that defines it in a different way than "missing"). nan is part of the IEEE floating point specification (as I'm sure you know) and it has a different meaning than null.

[pitrou]

I addressed some review comments now.

[pitrou]

AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.157

[pitrou]

Rebased.

[pitrou]

AppVeyor at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.175

[wesm]

+1, thanks for cleaning up the int/uint size issues here, much cleaner now

[wesm]

see ARROW-2298 for adding an option about NaN conversions