ARROW-2145/ARROW-2153/ARROW-2157/ARROW-2160/ARROW-2177: [Python] Decimal conversion not working for NaN values

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!

[cpcloud]

This PR closes the following JIRAs

ARROW-2145: [Python] Decimal conversion not working for NaN values
ARROW-2153: [C++/Python] Decimal conversion not working for exponential notation
ARROW-2157: [Python] Decimal arrays cannot be constructed from Python lists
ARROW-2160: [C++/Python] Fix decimal precision inference
ARROW-2177: [C++] Remove support for specifying negative scale values in DecimalType

I originally separated these fixes into a few smaller PRs, but it turned out
that the issues were all related, so I fixed them all in one PR.

[wesm]

Since we'll probably want to use libre2 for analytics, we should see at some point if we can replace the Boost regexen with libre2

[cpcloud]

@kou Do you have any idea why in this build: https://travis-ci.org/apache/arrow/jobs/345443821 OS X isn't finding the correct symbol? Is there some installation step for brew that I need to add?

Here's the error message:

dyld: Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
Referenced from: /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[cpcloud]

@wesm I'll open a JIRA for it.

[cpcloud]

Ah, looks like it was added in ARROW-29.

[cpcloud]

I'm not sure if this is the correct behavior here. I need to look into what other systems do with a string of all zeros for precision and scale.

[wesm]

This doesn't look like too much fun, thanks for slogging through this! left some stylistic comments and other things

[wesm]

I wonder if we should make some global state that is initialized when the library is loaded

[wesm]

aren't these dchecks already done?

[cpcloud]

Yep they are done in the Import* functions. I'll remove these.

[cpcloud]

I kept these DCHECKS since these functions are returning Status but I removed the messages.

[wesm]

what's the rationale for this, the symbol linking issue?

[cpcloud]

import pyarrow.parquet was segfaulting, I assumed because we're statically linking boost in the parquet build and dynamically in the arrow build. This only shows up when using the regex library.

[wesm]

I see, we should be consistent about which we do across the libraries. Part of why I wish we were building all these libraries in a monorepo setting

[wesm]

Ugh, Python, what did we do to deserve this? =)

[wesm]

these dchecks are performed twice -- should this be just DCHECK_OK on each of these?

[cpcloud]

I introduced a DCHECK_OK macro and used it here and in a few other places.

[wesm]

This should never happen by design, right?

[cpcloud]

Yep, I was guarding against potential uses of it after the fact so that arrow crashes with a useful error message to the developer.

[cpcloud]

I suppose I could relax this and just do nothing if the value is nan.

[wesm]

is there some TMP magic that makes this abstraction zero-cost, or does this add overhead?

[cpcloud]

So, operator[](const std::string) returns a const_reference to a sub_match object, which has a cast to std::string operator defined. sub_match has first and second attributes which are bidirectional iterators which are used to construct a string like std::string(match.first, match.second). Alternatively we use results["SIGN"].str(). The main difference is that the first uses __builtin_memcpy and the second uses reserve then ultimately __builtin_memset N number of times. I suspect that one call to memcpy N bytes is cheaper than N calls to memset individual elements.

[wesm]

I reckon we'll want to replace this with libre2 at some point. it's also a lot faster than boost::regex http://lh3lh3.users.sourceforge.net/reb.shtml

[cpcloud]

Yep, I'll make a JIRA for it.

[wesm]

FWIW it's not necessary to use this NULLPTR macro outside headers I don't believe

[cpcloud]

Cool I'll fix

[cpcloud]

Done

[wesm]

can you add a test here with an explicit decimal type sufficient to accommodate the data?

[wesm]

same here

[cpcloud]

This should ignore nans

[cpcloud]

The Update method now ignores nans

[cpcloud]

I'm going to change to ignore nans

[kou]

Umm. I have never seen the error. I may not help you because I'don't have macOS.

What are the outputs of the followings?

% nm /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% nm /usr/local/opt/boost/lib/libboost_regex-mt.dylib
% strings /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib | grep boost
% strings /usr/local/opt/boost/lib/libboost_regex-mt.dylib | grep boost
% otool -L /Users/travis/build/apache/arrow/cpp-install/lib/libarrow.0.dylib
% otool -L /usr/local/opt/boost/lib/libboost_regex-mt.dylib

[pitrou]

Shouldn't that be conditioned on ARROW_CI_C_GLIB_AFFECTED?

[cpcloud]

Yes

[cpcloud]

@pitrou This is already conditioned on in .travis.yml just before this script is called. Is it really necessary to condition on it again?

[pitrou]

Not really, though given the filename it might be better to avoid further mistakes :-)

[cpcloud]

@wesm@pitrou this is passing on travis: https://travis-ci.org/cpcloud/arrow/builds/347872453

[wesm]

Sweet, here is the Appveyor build: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.587. Going to take a quick look through and then merge

[wesm]

+1, thanks @cpcloud!