ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693

Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom
javierluraschi:bugfix/mingw32
Closed

ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom
javierluraschi:bugfix/mingw32

Conversation

@javierluraschi

@javierluraschijavierluraschi commented Feb 18, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.

@jeroen

jeroen commented Feb 18, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?

@kou

kou commented Feb 19, 2019

Copy link
Copy Markdown
Member

We need #3675 to enable tests for MinGW build.

@kou

kou commented Feb 19, 2019

Copy link
Copy Markdown
Member

Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46

@kou

kou commented Feb 19, 2019

Copy link
Copy Markdown
Member

#3657 has been merged.

We will be able to build and run tests with the following change:

diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi

javierluraschi commented Feb 19, 2019

Copy link
Copy Markdown
ContributorAuthor

Some tests are now failing to build,

make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou

kou commented Feb 20, 2019

Copy link
Copy Markdown
Member

We need to install gflags package.

diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^

pitrou
pitrou reviewed Feb 20, 2019
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.

Comment threadcpp/src/arrow/util/sse-util.h Outdated

@pitroupitrouFeb 20, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.

@javierluraschi
javierluraschiforce-pushed the bugfix/mingw32 branch 2 times, most recently from 4b2c569 to c60b966CompareFebruary 27, 2019 04:25
@javierluraschi

javierluraschi commented Feb 27, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?

@pitrou

pitrou commented Feb 27, 2019

Copy link
Copy Markdown
Member

@javierluraschi Indeed, they should be unrelated.

pitrou
pitrou reviewed Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
return 0;

@pitroupitrouFeb 27, 2019•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).

@javierluraschijavierluraschiFeb 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.

@javierluraschijavierluraschiFeb 28, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:

[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit

Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.

kou
kou reviewed Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
-DARROW_BUILD_TESTS=ON ^
-DARROW_PYTHON=OFF ^

@koukouFeb 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?
Could you show details of the error by resolving this?

@javierluraschijavierluraschiFeb 27, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.
The following patch will solve this.

diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%

@pitroupitrouFeb 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.

@javierluraschijavierluraschiFeb 28, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...

@javierluraschijavierluraschiMar 1, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:

[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:

> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:

> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.

@koukouMar 1, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?

kou
kou reviewed Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat Outdated

@koukouFeb 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.
The following command line is better because it shows details on error.

ctest --output-on-failure -j2 ||exit /B

kou
kou reviewed Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt Outdated

@koukouFeb 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?
Could you show error details?

@javierluraschijavierluraschiFeb 28, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.

@javierluraschijavierluraschiFeb 28, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:

/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.

@javierluraschijavierluraschiFeb 28, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.

@koukouMar 1, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.
We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.

@koukouMar 1, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.

@javierluraschi

javierluraschi commented Mar 1, 2019

Copy link
Copy Markdown
ContributorAuthor

@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!

@kou

kou commented Mar 1, 2019

Copy link
Copy Markdown
Member

We can merge this by opening JIRA issues for remaining tasks.

@javierluraschi

javierluraschi commented Mar 1, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@kou thank you! Here are the JIRA issues that would complete this PR:

ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds

@kou

kou commented Mar 1, 2019

Copy link
Copy Markdown
Member

Thanks!
I'll merge this.

@koukou closed this in d4931c9Mar 1, 2019
This was referenced Mar 1, 2019
Closed
Closed
@javierluraschijavierluraschi mentioned this pull request Mar 6, 2019
Merged
pitrou
pitrou reviewed Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
array-list-test.cc
array-struct-test.cc)

if(WIN32)

@pitroupitrouApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.
(also, this test should really be more narrow)

@koukouApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.
(I should have pushed a change for #3693 (comment) by myself.)

@asfimportasfimport mentioned this pull request Apr 3, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit - #3693
Closed
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Closed
ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit#3693
javierluraschi wants to merge 16 commits into
apache:masterfrom 
javierluraschi:bugfix/mingw32
Conversation
@javierluraschi
@javierluraschijavierluraschi
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.
@jeroen
jeroen
commented
Feb 18, 2019•

edited

Loading

Copy link


Copy Markdown

Contributor
It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
We need #3675 to enable tests for MinGW build.
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46
@kou
kou
commented
Feb 19, 2019

Copy link


Copy Markdown

Member
#3657 has been merged.
We will be able to build and run tests with the following change:
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -44,8 +44,10 @@ cmake ^
-DARROW_PYTHON=ON ^
-DPythonInterp_FIND_VERSION=ON ^
-DPythonInterp_FIND_VERSION_MAJOR=3 ^
+ -DARROW_BUILD_TESTS=ON ^
.. || exit /B
make -j4 || exit /B
+make test || exit /B
make install || exit /B
popd

@javierluraschi
javierluraschi
commented
Feb 19, 2019

Copy link


Copy Markdown

ContributorAuthor
Some tests are now failing to build,
make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'. Stop.

@kou
kou
commented
Feb 20, 2019

Copy link


Copy Markdown

Member
We need to install gflags package.
diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat+++ b/ci/appveyor-cpp-setup-mingw.bat@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
"%MINGW_PACKAGE_PREFIX%-cmake" ^
"%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
"%MINGW_PACKAGE_PREFIX%-gcc" ^
+ "%MINGW_PACKAGE_PREFIX%-gflags" ^
"%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
"%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
"%MINGW_PACKAGE_PREFIX%-lz4" ^
pitrou
pitrou
reviewed
Feb 20, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.
Comment threadcpp/src/arrow/util/sse-util.h
Outdated

@pitroupitrouFeb 20, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.
@javierluraschi
javierluraschiforce-pushed
the
bugfix/mingw32
branch
2 times, most recently
from
4b2c569 to
c60b966CompareFebruary 27, 2019 04:25
@javierluraschi
javierluraschi
commented
Feb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?
@pitrou
pitrou
commented
Feb 27, 2019

Copy link


Copy Markdown

Member
@javierluraschi Indeed, they should be unrelated.
pitrou
pitrou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadcpp/src/arrow/util/sse-util.h

static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
#if ARROW_BITNESS == 32
 return 0;
@pitroupitrouFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).
@javierluraschijavierluraschiFeb 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aborting with std::abort.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this introduced the following build error, so reverted the change:
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 44%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit 
Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.
kou
kou
reviewed
Feb 27, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
 -DPythonInterp_FIND_VERSION=ON ^
 -DPythonInterp_FIND_VERSION_MAJOR=3 ^
 -DARROW_BUILD_TESTS=ON ^
 -DARROW_PYTHON=OFF ^
@koukouFeb 27, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Could you show details of the error by resolving this?
@javierluraschijavierluraschiFeb 27, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou, sure:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

The following patch will solve this.
diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 06e8b7f7..3a853031 100644
--- a/ci/appveyor-cpp-build-mingw.bat+++ b/ci/appveyor-cpp-build-mingw.bat@@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install
set PATH=%INSTALL_DIR%\bin;%PATH%
set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig
+for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do (+ set PYTHON_VERSION=%%v+)++set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION%+set PYTHONPATH=%PYTHONHOME%+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload+set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages+
set CPP_BUILD_DIR=cpp\build
mkdir %CPP_BUILD_DIR%
pushd %CPP_BUILD_DIR%
@pitroupitrouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try that, give me a sec...
@javierluraschijavierluraschiMar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj
[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj
[ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj
[ 45%] Built target arrow_objlib
make: *** [Makefile:141: all] Error 2
C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't resolve without PYTHONPATH on local:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll
# trying C:/msys64/mingw64/bin\encodings-abi3.dll
# trying C:/msys64/mingw64/bin\encodings.cp37.pyd
# trying C:/msys64/mingw64/bin\encodings.pyd
# trying C:/msys64/mingw64/bin\encodings.py
# trying C:/msys64/mingw64/bin\encodings.pyw
# trying C:/msys64/mingw64/bin\encodings.pyc
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
# trying C:/msys64/mingw64/bin\io-cpython-37m.dll
# trying C:/msys64/mingw64/bin\io-abi3.dll
# trying C:/msys64/mingw64/bin\io.cp37.pyd
# trying C:/msys64/mingw64/bin\io.pyd
# trying C:/msys64/mingw64/bin\io.py
# trying C:/msys64/mingw64/bin\io.pyw
# trying C:/msys64/mingw64/bin\io.pyc
ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:
> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7
> set PYTHONPATH=%PYTHONHOME%
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload
> set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages
> python3 -vv -c 1
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.pyd
# trying c:\msys64\mingw64\lib\python3.7\codecs.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080>
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8>
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd
# trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80>
# trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.pyd
# trying c:\msys64\mingw64\lib\python3.7\io.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0>
# trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.pyd
# trying c:\msys64\mingw64\lib\python3.7\site.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.pyd
# trying c:\msys64\mingw64\lib\python3.7\os.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.pyd
# trying c:\msys64\mingw64\lib\python3.7\stat.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8>
# trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\ntpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc'
# trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd
# trying c:\msys64\mingw64\lib\python3.7\genericpath.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10>
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd
# trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8>
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd
# trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py
# code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50>
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd
# trying C:/msys64/mingw64/lib/python3.7\sysconfig.py
# C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py
# code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0>
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc
# trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll
# trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.pyd
# trying C:/msys64/mingw64/bin\sitecustomize.py
# trying C:/msys64/mingw64/bin\sitecustomize.pyw
# trying C:/msys64/mingw64/bin\sitecustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw
# trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc
# trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll
# trying C:/msys64/mingw64/bin\usercustomize-abi3.dll
# trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd
# trying C:/msys64/mingw64/bin\usercustomize.pyd
# trying C:/msys64/mingw64/bin\usercustomize.py
# trying C:/msys64/mingw64/bin\usercustomize.pyw
# trying C:/msys64/mingw64/bin\usercustomize.pyc
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60>
Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# clear builtins._
...

@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to set environment variables for Python only around testing.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to work this on another pull request, could you open an issue on JIRA?
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadci/appveyor-cpp-build-mingw.bat
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry.

The following command line is better because it shows details on error.
ctest --output-on-failure -j2 ||exit /B
kou
kou
reviewed
Feb 28, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

@koukouFeb 28, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it needed?

Could you show error details?
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.
@javierluraschijavierluraschiFeb 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, so from AppVeyor, looks like this is causing:
/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)'
CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.
@javierluraschijavierluraschiFeb 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.
@koukouMar 1, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.
@javierluraschi
javierluraschi
commented
Mar 1, 2019

Copy link


Copy Markdown

ContributorAuthor
@kou@wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
We can merge this by opening JIRA issues for remaining tasks.
@javierluraschi
javierluraschi
commented
Mar 1, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou thank you! Here are the JIRA issues that would complete this PR:
ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds
@kou
kou
commented
Mar 1, 2019

Copy link


Copy Markdown

Member
Thanks!

I'll merge this.
@koukou
closed this
in
d4931c9Mar 1, 2019

This was referenced Mar 1, 2019
 Closed

 Closed

@javierluraschijavierluraschi
mentioned this pull request
Mar 6, 2019
 Merged

pitrou
pitrou
reviewed
Apr 3, 2019
View reviewed changes
Comment threadcpp/src/arrow/CMakeLists.txt
 array-list-test.cc
 array-struct-test.cc)

if(WIN32)
@pitroupitrouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javierluraschi@kou In the future, please add a comment when you add a special case like this, so that we know why it is there.

(also, this test should really be more narrow)
@koukouApr 3, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Sorry for this.

(I should have pushed a change for #3693 (comment) by myself.)
@asfimportasfimport
mentioned this pull request
Apr 3, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou left review comments
@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@javierluraschi@jeroen@kou@pitrou