ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit #3693

javierluraschi · 2019-02-18T19:06:55Z

Improvements for ARROW-4297. This PR is a follow up to the discussed patch from #3443.

jeroen · 2019-02-18T22:32:54Z

It seems to build now on mingw32 👍 Is there a test that checks if it actually gives the correct results ?

kou · 2019-02-19T03:47:38Z

We need #3675 to enable tests for MinGW build.

kou · 2019-02-19T03:48:33Z

Could you remove allow_failures?
https://github.com/apache/arrow/blob/master/appveyor.yml#L39-L46

kou · 2019-02-19T08:17:21Z

#3657 has been merged.

We will be able to build and run tests with the following change:

diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat
index 4d399274..d70fce1a 100644
--- a/ci/appveyor-cpp-build-mingw.bat
+++ b/ci/appveyor-cpp-build-mingw.bat
@@ -44,8 +44,10 @@ cmake ^
     -DARROW_PYTHON=ON ^
     -DPythonInterp_FIND_VERSION=ON ^
     -DPythonInterp_FIND_VERSION_MAJOR=3 ^
+    -DARROW_BUILD_TESTS=ON ^
     .. || exit /B
 make -j4 || exit /B
+make test || exit /B
 make install || exit /B
 popd

javierluraschi · 2019-02-19T21:26:42Z

Some tests are now failing to build,

make[2]: *** No rule to make target 'gflags_ep-prefix/src/gflags_ep/lib/libgflags.a',
needed by 'release/arrow-json-integration-test.exe'.  Stop.

kou · 2019-02-20T00:31:24Z

We need to install gflags package.

diff --git a/ci/appveyor-cpp-setup-mingw.bat b/ci/appveyor-cpp-setup-mingw.bat
index 471e7426..0c3d633c 100644
--- a/ci/appveyor-cpp-setup-mingw.bat
+++ b/ci/appveyor-cpp-setup-mingw.bat
@@ -25,6 +25,7 @@ pacman -S --noconfirm ^
     "%MINGW_PACKAGE_PREFIX%-cmake" ^
     "%MINGW_PACKAGE_PREFIX%-flatbuffers" ^
     "%MINGW_PACKAGE_PREFIX%-gcc" ^
+    "%MINGW_PACKAGE_PREFIX%-gflags" ^
     "%MINGW_PACKAGE_PREFIX%-gobject-introspection" ^
     "%MINGW_PACKAGE_PREFIX%-gtk-doc" ^
     "%MINGW_PACKAGE_PREFIX%-lz4" ^

pitrou

LGTM on the principle. You'll need to fix the CI problems on AppVeyor :-) Also one comment.

pitrou · 2019-02-20T10:03:41Z

cpp/src/arrow/util/sse-util.h

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.

javierluraschi · 2019-02-27T19:31:15Z

@kou @wesm the nodejs and python travis errors seem unrelated, are they currently failing for other builds?

pitrou · 2019-02-27T19:38:32Z

@javierluraschi Indeed, they should be unrelated.

pitrou · 2019-02-27T19:39:39Z

cpp/src/arrow/util/sse-util.h


 static inline uint32_t SSE4_crc32_u64(uint32_t crc, uint64_t v) {
+#if ARROW_BITNESS == 32
+  return 0;


I would expect a std::abort call or equivalent here (you can also use ARROW_LOG(FATAL)).

Aborting with std::abort.

Looks like this introduced the following build error, so reverted the change:

[ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj [ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj [ 44%] Built target arrow_objlib make: *** [Makefile:141: all] Error 2 C:\projects\arrow\cpp\build>goto scriptexit

Unfortunately, this build error does not reproduce in my local environment and further information is not available in appveyor.

kou · 2019-02-27T20:04:28Z

ci/appveyor-cpp-build-mingw.bat

    -DPythonInterp_FIND_VERSION=ON ^
    -DPythonInterp_FIND_VERSION_MAJOR=3 ^
+    -DARROW_BUILD_TESTS=ON ^
+    -DARROW_PYTHON=OFF ^


Why is this needed?
Could you show details of the error by resolving this?

@kou, sure:

Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] Fatal Python error: Py_Initialize: unable to load the file system codec ModuleNotFoundError: No module named 'encodings'

Thanks.
The following patch will solve this.

diff --git a/ci/appveyor-cpp-build-mingw.bat b/ci/appveyor-cpp-build-mingw.bat index 06e8b7f7..3a853031 100644 --- a/ci/appveyor-cpp-build-mingw.bat +++ b/ci/appveyor-cpp-build-mingw.bat @@ -24,6 +24,15 @@ set INSTALL_DIR=%HOMEDRIVE%%HOMEPATH%\install set PATH=%INSTALL_DIR%\bin;%PATH% set PKG_CONFIG_PATH=%INSTALL_DIR%\lib\pkgconfig +for /f "usebackq" %%v in (`python3 -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))"`) do ( + set PYTHON_VERSION=%%v +) + +set PYTHONHOME=%MINGW_PREFIX%\lib\python%PYTHON_VERSION% +set PYTHONPATH=%PYTHONHOME% +set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\lib-dynload +set PYTHONPATH=%PYTHONPATH%;%MINGW_PREFIX%\lib\python%PYTHON_VERSION%\site-packages + set CPP_BUILD_DIR=cpp\build mkdir %CPP_BUILD_DIR% pushd %CPP_BUILD_DIR%

If you set PYTHONHOME you shouldn't have to set PYTHONPATH as well. Python will infer the path automatically from PYTHONHOME.

I will try that, give me a sec...

Actually, the original suggestion breaks the build, see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22735113/job/y7dijp1ym0y1q87g, which triggers:

[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.obj [ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.obj [ 44%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.obj [ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/reader.cc.obj [ 45%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.obj [ 45%] Built target arrow_objlib make: *** [Makefile:141: all] Error 2 C:\projects\arrow\cpp\build>goto scriptexit C:\projects\arrow\cpp\build>set lastexitcode=2 C:\projects\arrow\cpp\build>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B0.tmp C:\projects\arrow\cpp\build>echo C:\projects\arrow\cpp\build 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp56B1.tmp C:\projects\arrow\cpp\build>exit /b 2

I couldn't resolve without PYTHONPATH on local:

> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7 > python3 -vv -c 1 import _frozen_importlib # frozen import _imp # builtin import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import '_warnings' # <class '_frozen_importlib.BuiltinImporter'> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # installing zipimport hook import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'> # installed zipimport hook import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'> import '_io' # <class '_frozen_importlib.BuiltinImporter'> import 'marshal' # <class '_frozen_importlib.BuiltinImporter'> import 'nt' # <class '_frozen_importlib.BuiltinImporter'> import _thread # previously loaded ('_thread') import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import _weakref # previously loaded ('_weakref') import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> import 'winreg' # <class '_frozen_importlib.BuiltinImporter'> # trying C:/msys64/mingw64/bin\encodings-cpython-37m.dll # trying C:/msys64/mingw64/bin\encodings-abi3.dll # trying C:/msys64/mingw64/bin\encodings.cp37.pyd # trying C:/msys64/mingw64/bin\encodings.pyd # trying C:/msys64/mingw64/bin\encodings.py # trying C:/msys64/mingw64/bin\encodings.pyw # trying C:/msys64/mingw64/bin\encodings.pyc Fatal Python error: initfsencoding: unable to load the file system codec Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 983, in _find_and_load # trying C:/msys64/mingw64/bin\io-cpython-37m.dll # trying C:/msys64/mingw64/bin\io-abi3.dll # trying C:/msys64/mingw64/bin\io.cp37.pyd # trying C:/msys64/mingw64/bin\io.pyd # trying C:/msys64/mingw64/bin\io.py # trying C:/msys64/mingw64/bin\io.pyw # trying C:/msys64/mingw64/bin\io.pyc File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked # trying C:/msys64/mingw64/bin\io-cpython-37m.dll # trying C:/msys64/mingw64/bin\io-abi3.dll # trying C:/msys64/mingw64/bin\io.cp37.pyd # trying C:/msys64/mingw64/bin\io.pyd # trying C:/msys64/mingw64/bin\io.py # trying C:/msys64/mingw64/bin\io.pyw # trying C:/msys64/mingw64/bin\io.pyc ModuleNotFoundError: No module named 'encodings'

With PYTHONPATH:

> set PYTHONHOME=c:\msys64\mingw64\lib\python3.7 > set PYTHONPATH=%PYTHONHOME% > set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\lib-dynload > set PYTHONPATH=%PYTHONPATH%;c:\msys64\mingw64\lib\python3.7\site-packages > python3 -vv -c 1 import _frozen_importlib # frozen import _imp # builtin import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import '_warnings' # <class '_frozen_importlib.BuiltinImporter'> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # installing zipimport hook import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'> # installed zipimport hook import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'> import '_io' # <class '_frozen_importlib.BuiltinImporter'> import 'marshal' # <class '_frozen_importlib.BuiltinImporter'> import 'nt' # <class '_frozen_importlib.BuiltinImporter'> import _thread # previously loaded ('_thread') import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import _weakref # previously loaded ('_weakref') import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> import 'winreg' # <class '_frozen_importlib.BuiltinImporter'> # c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\__init__.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\__init__.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\__init__.cpython-37.pyc' # trying c:\msys64\mingw64\lib\python3.7\codecs-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\codecs-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\codecs.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\codecs.pyd # trying c:\msys64\mingw64\lib\python3.7\codecs.py # c:\msys64\mingw64\lib\python3.7\__pycache__\codecs.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\codecs.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\codecs.cpython-37.pyc' import '_codecs' # <class '_frozen_importlib.BuiltinImporter'> import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029fe080> # trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\aliases-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\aliases.py # c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\aliases.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\aliases.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\aliases.cpython-37.pyc' import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a099b0> import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x00000000029f02e8> # trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py # c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\utf_8.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\utf_8.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\utf_8.cpython-37.pyc' import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1aa58> import '_signal' # <class '_frozen_importlib.BuiltinImporter'> # trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.pyd # trying c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py # c:\msys64\mingw64\lib\python3.7\encodings\__pycache__\latin_1.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\encodings\latin_1.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\encodings\\__pycache__\\latin_1.cpython-37.pyc' import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ae80> # trying c:\msys64\mingw64\lib\python3.7\io-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\io-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\io.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\io.pyd # trying c:\msys64\mingw64\lib\python3.7\io.py # c:\msys64\mingw64\lib\python3.7\__pycache__\io.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\io.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\io.cpython-37.pyc' # trying c:\msys64\mingw64\lib\python3.7\abc-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\abc-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\abc.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\abc.pyd # trying c:\msys64\mingw64\lib\python3.7\abc.py # c:\msys64\mingw64\lib\python3.7\__pycache__\abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\abc.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\abc.cpython-37.pyc' import '_abc' # <class '_frozen_importlib.BuiltinImporter'> import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e4a8> import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1e0f0> # trying c:\msys64\mingw64\lib\python3.7\site-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\site-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\site.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\site.pyd # trying c:\msys64\mingw64\lib\python3.7\site.py # c:\msys64\mingw64\lib\python3.7\__pycache__\site.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\site.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\site.cpython-37.pyc' # trying c:\msys64\mingw64\lib\python3.7\os-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\os-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\os.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\os.pyd # trying c:\msys64\mingw64\lib\python3.7\os.py # c:\msys64\mingw64\lib\python3.7\__pycache__\os.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\os.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\os.cpython-37.pyc' # trying c:\msys64\mingw64\lib\python3.7\stat-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\stat-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\stat.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\stat.pyd # trying c:\msys64\mingw64\lib\python3.7\stat.py # c:\msys64\mingw64\lib\python3.7\__pycache__\stat.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\stat.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\stat.cpython-37.pyc' import '_stat' # <class '_frozen_importlib.BuiltinImporter'> import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1a7b8> # trying c:\msys64\mingw64\lib\python3.7\ntpath-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\ntpath-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\ntpath.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\ntpath.pyd # trying c:\msys64\mingw64\lib\python3.7\ntpath.py # c:\msys64\mingw64\lib\python3.7\__pycache__\ntpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\ntpath.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\ntpath.cpython-37.pyc' # trying c:\msys64\mingw64\lib\python3.7\genericpath-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\genericpath-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\genericpath.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\genericpath.pyd # trying c:\msys64\mingw64\lib\python3.7\genericpath.py # c:\msys64\mingw64\lib\python3.7\__pycache__\genericpath.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\genericpath.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\genericpath.cpython-37.pyc' import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b278d0> import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b1ae10> # trying c:\msys64\mingw64\lib\python3.7\_collections_abc-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\_collections_abc-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\_collections_abc.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\_collections_abc.pyd # trying c:\msys64\mingw64\lib\python3.7\_collections_abc.py # c:\msys64\mingw64\lib\python3.7\__pycache__\_collections_abc.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_collections_abc.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_collections_abc.cpython-37.pyc' import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002b27e10> import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2e3c8> # trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-cpython-37m.dll # trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins-abi3.dll # trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.cp37.pyd # trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.pyd # trying c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py # c:\msys64\mingw64\lib\python3.7\__pycache__\_sitebuiltins.cpython-37.pyc matches c:\msys64\mingw64\lib\python3.7\_sitebuiltins.py # code object from 'c:\\msys64\\mingw64\\lib\\python3.7\\__pycache__\\_sitebuiltins.cpython-37.pyc' import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2ec50> # trying C:/msys64/mingw64/lib/python3.7\sysconfig-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7\sysconfig-abi3.dll # trying C:/msys64/mingw64/lib/python3.7\sysconfig.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7\sysconfig.pyd # trying C:/msys64/mingw64/lib/python3.7\sysconfig.py # C:/msys64/mingw64/lib/python3.7\__pycache__\sysconfig.cpython-37.pyc matches C:/msys64/mingw64/lib/python3.7\sysconfig.py # code object from 'C:/msys64/mingw64/lib/python3.7\\__pycache__\\sysconfig.cpython-37.pyc' import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a2eda0> # trying C:/msys64/mingw64/lib/python3.7\sitecustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7\sitecustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7\sitecustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyd # trying C:/msys64/mingw64/lib/python3.7\sitecustomize.py # trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyw # trying C:/msys64/mingw64/lib/python3.7\sitecustomize.pyc # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyd # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.py # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyw # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\sitecustomize.pyc # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyd # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.py # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyw # trying C:/msys64/mingw64/lib/python3.7/site-packages\sitecustomize.pyc # trying C:/msys64/mingw64/bin\sitecustomize-cpython-37m.dll # trying C:/msys64/mingw64/bin\sitecustomize-abi3.dll # trying C:/msys64/mingw64/bin\sitecustomize.cp37.pyd # trying C:/msys64/mingw64/bin\sitecustomize.pyd # trying C:/msys64/mingw64/bin\sitecustomize.py # trying C:/msys64/mingw64/bin\sitecustomize.pyw # trying C:/msys64/mingw64/bin\sitecustomize.pyc # trying C:/msys64/mingw64/lib/python3.7\usercustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7\usercustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7\usercustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyd # trying C:/msys64/mingw64/lib/python3.7\usercustomize.py # trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyw # trying C:/msys64/mingw64/lib/python3.7\usercustomize.pyc # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyd # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.py # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyw # trying C:/msys64/mingw64/lib/python3.7/lib-dynload\usercustomize.pyc # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-cpython-37m.dll # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize-abi3.dll # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.cp37.pyd # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyd # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.py # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyw # trying C:/msys64/mingw64/lib/python3.7/site-packages\usercustomize.pyc # trying C:/msys64/mingw64/bin\usercustomize-cpython-37m.dll # trying C:/msys64/mingw64/bin\usercustomize-abi3.dll # trying C:/msys64/mingw64/bin\usercustomize.cp37.pyd # trying C:/msys64/mingw64/bin\usercustomize.pyd # trying C:/msys64/mingw64/bin\usercustomize.py # trying C:/msys64/mingw64/bin\usercustomize.pyw # trying C:/msys64/mingw64/bin\usercustomize.pyc import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0000000002a1ef60> Python 3.7.0 (default, Aug 3 2018, 11:56:18) [GCC 8.2.0 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. # clear builtins._ ...

We may need to set environment variables for Python only around testing.

If we want to work this on another pull request, could you open an issue on JIRA?

kou · 2019-02-28T00:29:14Z

ci/appveyor-cpp-build-mingw.bat

Sorry.
The following command line is better because it shows details on error.

ctest --output-on-failure -j2 || exit /B

kou · 2019-02-28T00:30:03Z

cpp/src/arrow/CMakeLists.txt

Why is it needed?
Could you show error details?

I don't know, I wasn't able to reproduce this one on my local MinGW 32-bit environment, but let me disable it and hopefully your new test command will output something in appveyor that can point us in the right direction.

Alright, so from AppVeyor, looks like this is causing:

/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xb9a2): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)' CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xcb8a): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)' CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0xeef8): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)' CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x10240): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::DictionaryBuilder(std::shared_ptr<arrow::DataType> const&, arrow::MemoryPool*)' CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x104fc): undefined reference to `arrow::DictionaryBuilder<arrow::FixedSizeBinaryType>::AppendArray(arrow::Array const&)' CMakeFiles/arrow-array-test.dir/objects.a(array-dict-test.cc.obj):array-dict-test.cc:(.text+0x108ef): undefined reference to `arrow

Which again, I can't repro locally.

Let me re-skip this test to have them back at passing... My suggestion would be to merge with a couple tests skipped and then open JIRA issues to investigate further. Otherwise, the longer it take us to merge this PR, the more likely something else will break Windows.

Thanks.
We need to stop using template and ARROW_EXPORT for arrow::DictionaryBuilder like #3509.

Could you open an issue on JIRA to work this on another pull request?
if(WIN32) is too general. It matches build with MSVC.
if(WIN32 AND "${COMPILER_FAMIL}" STREQUAL "gcc") is better.

This reverts commit f2e4dfa.

This reverts commit e69b1e9.

This reverts commit 2ee894f.

javierluraschi · 2019-03-01T04:00:15Z

@kou @wesm could we merge this PR as is? While there are additional suggestions, like enabling Python tests, troubleshooting AppVeyor can be time consuming when a local environment does not reproduce the remote environment. Besides, I believe starting with a subset of the tests for a new build flavor is in general acceptable. Thanks!

kou · 2019-03-01T05:19:25Z

We can merge this by opening JIRA issues for remaining tasks.

javierluraschi · 2019-03-01T05:39:19Z

@kou thank you! Here are the JIRA issues that would complete this PR:

ARROW-4724: [C++] Python not being built nor test under MinGW builds
ARROW-4725: [C++] Dictionary tests disabled under MinGW builds
ARROW-4726: [C++] IntToFloatingPoint tests disabled under 32bit builds

kou · 2019-03-01T06:21:23Z

Thanks!
I'll merge this.

pitrou · 2019-04-03T12:32:23Z

cpp/src/arrow/CMakeLists.txt

-               array-list-test.cc
-               array-struct-test.cc)
+
+if(WIN32)


@javierluraschi @kou In the future, please add a comment when you add a special case like this, so that we know why it is there.
(also, this test should really be more narrow)

OK. Sorry for this.
(I should have pushed a change for #3693 (comment) by myself.)

wesm force-pushed the master branch from 3088183 to 0c6b2d2 Compare February 18, 2019 19:34

javierluraschi force-pushed the bugfix/mingw32 branch from 9c367ee to 2f6e1b6 Compare February 19, 2019 16:20

pitrou reviewed Feb 20, 2019

View reviewed changes

cpp/src/arrow/util/sse-util.h Outdated

Copy link

Member

pitrou Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's MinGW-specific, so you could use ARROW_BITNESS here.

javierluraschi force-pushed the bugfix/mingw32 branch 2 times, most recently from 4b2c569 to c60b966 Compare February 27, 2019 04:25

kou and others added 9 commits February 27, 2019 09:52

[C++] Fix build error with MinGW-w64 32-bit

219384f

[C++] Apply patch discussed under pull/3443

aad43b2

[C++] Add MinGW32 tests as suggested in pull/3693

695bdc9

[C++] Remove allow_failures as suggested in pull/3693

a24285b

[C++] Install gflags as suggested in pull/3693

057347a

[C++] skip array-dict-test in win32 builds

9c1d6e4

[C++] Use ARROW_BITNESS instead of MinGW checks

3a0620f

[C++] Skip failing tests to investigate later

e0058b2

[C++] Skip python since not installed in appveyor

5b904fa

javierluraschi force-pushed the bugfix/mingw32 branch from 560ed46 to 5b904fa Compare February 27, 2019 17:53

pitrou reviewed Feb 27, 2019

View reviewed changes

kou reviewed Feb 27, 2019

View reviewed changes

javierluraschi force-pushed the bugfix/mingw32 branch from 3b38c8f to cb4f95d Compare February 27, 2019 23:18

kou reviewed Feb 28, 2019

View reviewed changes

javierluraschi added 2 commits February 27, 2019 21:16

[C++] Show error details in test as suggested in pull/3693

4cb63b1

[C++] Install python as suggested in pull/3693

aa9f711

[C++] Revert disabled test to investigate appveyor failure

f2e4dfa

javierluraschi force-pushed the bugfix/mingw32 branch from 31076f2 to f2e4dfa Compare February 28, 2019 05:17

javierluraschi added 4 commits February 28, 2019 07:34

[C++] Revert install python as suggested in pull/3693

e69b1e9

Revert "[C++] Revert disabled test to investigate appveyor failure"

41ae266

This reverts commit f2e4dfa.

Revert "[C++] Revert install python as suggested in pull/3693"

2ee894f

This reverts commit e69b1e9.

Revert "Revert "[C++] Revert install python as suggested in pull/3693""

e56dd14

This reverts commit 2ee894f.

kou closed this in d4931c9 Mar 1, 2019

This was referenced Mar 1, 2019

ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit #3443

Closed

ARROW-4744: [C++][CI] Change mingw builds back to debug. Cleanup up some version warnings #3793

Closed

javierluraschi mentioned this pull request Mar 6, 2019

Attempt to use arrow 0.13 devel branch r-windows/rtools-packages#11

Merged

pitrou reviewed Apr 3, 2019

View reviewed changes

asfimport mentioned this pull request Apr 3, 2019

[C++] Fix build for 32-bit MSYS2 #20870

Closed

Uh oh!

ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit #3693

ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit #3693

Uh oh!

Conversation

javierluraschi commented Feb 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeroen commented Feb 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Feb 19, 2019

Uh oh!

kou commented Feb 19, 2019

Uh oh!

kou commented Feb 19, 2019

Uh oh!

javierluraschi commented Feb 19, 2019

Uh oh!

kou commented Feb 20, 2019

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Feb 27, 2019

Uh oh!

pitrou Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javierluraschi commented Mar 1, 2019

javierluraschi commented Feb 18, 2019 •

edited

Loading

jeroen commented Feb 18, 2019 •

edited

Loading

javierluraschi commented Feb 27, 2019 •

edited

Loading

pitrou Feb 27, 2019 •

edited

Loading

javierluraschi Feb 28, 2019 •

edited

Loading

javierluraschi Feb 27, 2019 •

edited

Loading

javierluraschi Feb 28, 2019 •

edited

Loading

javierluraschi Feb 28, 2019 •

edited

Loading

javierluraschi commented Mar 1, 2019 •

edited

Loading