ARROW-3803: [C++/Python] Merge C++ builds and tests, run Python tests in separate CI entries

[wesm]

I found unfortunately that the conda-forge boost-cpp package is not fully compatible with Xcode 8.3, see https://issues.apache.org/jira/browse/ARROW-4056

We might have to build a vendored Boost in this CI entry to work around the problem (this is what the Ray project did when they also hit this issue)

[wesm]

We may be reaching the end of the road with Xcode 6.4. Travis is pulling support for the image in January 2019 according to messages in the build logs.

Gandiva does not run properly (due to a strptime bug?) on 6.4 so we might move everything to 8.3. We should see what conda-forge is planning exactly, since if they are remaining on 6.4 then we may not be able to have Gandiva in the Python wheels

cc @pravindra@praveenbingo

[pravindra]

@wesm current change lgtm i.e CI has gandiva with Xcode 8.3 but not Xcode 6.4.

If gandiva with Xcode 6.4 becomes important/required for any reason, we can spend some time on workarounds to make it work.

[wesm]

I opened https://issues.apache.org/jira/browse/ARROW-4068 about Xcode 6.4. If you know what problems there are, could you post them there? I'm working on getting a reliable macOS box where I can do development and testing

[wesm]

I ran into https://issues.apache.org/jira/browse/ARROW-4070 here when trying to use ARROW_BOOST_VENDORED with ninja

[fsaintjacques]

LD_LIBRARY_PATH is less invasive.

[wesm]

LD_LIBRARY_PATH is already set...

https://github.com/apache/arrow/blob/master/ci/travis_env_common.sh#L63

[wesm]

I think this is almost there. I'm just waiting on the Gandiva valgrind fixes to go in. If folks could take pity on me and try to not touch the CI scripts any more until this is merged that would be really great

[codecov-io]

Codecov Report

Merging #3208 into master will increase coverage by 20.58%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3208 +/- ##
==========================================
+ Coverage 67.62% 88.2% +20.58% 
==========================================
Files 58 550 +492 Lines 4012 73840 +69828 ==========================================
+ Hits 2713 65132 +62419 - Misses 1192 8601 +7409 
Partials 107 107

Impacted Files	Coverage Δ
python/pyarrow/ipc.pxi	67.53% <0%> (ø)
cpp/src/arrow/csv/chunker-test.cc	100% <0%> (ø)
cpp/src/parquet/column_page.h	89.28% <0%> (ø)
cpp/src/parquet/bloom_filter-test.cc	99.13% <0%> (ø)
cpp/src/arrow/array/builder_decimal.cc	100% <0%> (ø)
cpp/src/plasma/client.cc	96.02% <0%> (ø)
cpp/src/arrow/io/test-common.h	97.29% <0%> (ø)
cpp/src/gandiva/function_registry.h	100% <0%> (ø)
cpp/src/arrow/util/int-util-test.cc	100% <0%> (ø)
cpp/src/arrow/python/io.cc	95.72% <0%> (ø)
... and 482 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25b6a6c...32cffe0. Read the comment docs.

[pravindra]

I opened https://issues.apache.org/jira/browse/ARROW-4068 about Xcode 6.4. If you know what problems there are, could you post them there? I'm working on getting a reliable macOS box where I can do development and testing

will do. thanks.

[wesm]

Rebased. Will merge once the build runs

[wesm]

Running all the C++ unit tests under valgrind is taking about 23 minutes

Label Time Summary:
arrow-tests = 369.38 sec*proc (46 tests)
gandiva-tests = 575.75 sec*proc (27 tests)
parquet-tests = 458.77 sec*proc (18 tests)
plasma-tests = 14.58 sec*proc (2 tests)
unittest = 1418.48 sec*proc (93 tests)

This is OK for now, but may not be scalable.

[wesm]

Overall this patch cuts about 10 minutes out of build times

Will merge as soon as Appveyor is looking ok

[wesm]

The Appveyor builds passed (Rust is still running -- I opened https://issues.apache.org/jira/browse/ARROW-4080)