Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bolt Python wrapper doesn't build on some systems #4

Open
dblalock opened this issue Jul 5, 2017 · 24 comments
Open

Bolt Python wrapper doesn't build on some systems #4

dblalock opened this issue Jul 5, 2017 · 24 comments
Assignees
Labels

Comments

@dblalock
Copy link
Owner

dblalock commented Jul 5, 2017

At present, Bolt's python code doesn't appear to be building on Ubuntu and hasn't been tested on Windows. It may also have issues building on other systems.

@dblalock dblalock self-assigned this Jul 5, 2017
@arminbashizade
Copy link

arminbashizade commented Jan 6, 2018

I could build the code on redhat with this modifications:

  • include stdint.h in cpp/src/quantize/bolt.hpp
  • use swig 3.0
  • compile with clang 3.9 (in setup.py: extra_args = ['std=c++1y', '-stdlib=libc++', ...] and os.environ['CC'] = "clang" and os.environ['CXX'] = "clang++")

now my problem is when I try to import bolt I get this error:

>>> import bolt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/__init__.py", line 5, in <module>
    from bolt_api import *  # noqa
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt_api.py", line 6, in <module>
    import bolt  # inner bolt because of SWIG
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt.py", line 17, in <module>
    _bolt = swig_import_helper()
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt.py", line 16, in swig_import_helper
    return importlib.import_module('_bolt')
  File "~/anaconda2/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: ~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/_bolt.so: undefined symbol: _ZNKSt3__120__vector_base_commonILb1EE20__throw_length_errorEv

@arminbashizade
Copy link

setting the environment variable LDFLAGS to -lc++ solved the problem. (adding os.environ['LDFLAGS'] = '-lc++' to setup.py)

@dblalock
Copy link
Owner Author

I marked this as closed by adding your changes as per our email exchange. Thanks for your help in getting to the bottom of this. For those seeing this in the future, I'm not 100% certain this fix will work on all systems and so would be happy to reopen this if you have problems.

@goobee
Copy link

goobee commented Sep 5, 2018

I miss another problem look like this
python/bolt/../../cpp/src/include/public.hpp:23: Warning 315: Nothing known about 'std::vector'.
python/bolt/config.i:45: Warning 315: Nothing known about 'Eigen::Matrix'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:47: Warning 315: Nothing known about 'Eigen::RowMajor'.
python/bolt/config.i:45: Warning 315: Nothing known about 'Eigen::Matrix'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:47: Warning 315: Nothing known about 'Eigen::RowMajor'.
clang -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNDEBUG=1 -I/home/public/anaconda3/lib/python3.6/site-packages/numpy/core/include -Icpp/src/quantize -Icpp/src/utils -Icpp/src/include -Icpp/src/external -Icpp/src/external/eigen -I/home/yhx/bolt/python/bolt -I/home/public/anaconda3/include/python3.6m -c python/bolt/native_wrap.cpp -o build/temp.linux-x86_64-3.6/python/bolt/native_wrap.o -std=c++14 -fno-rtti -march=native -ffast-math -std=c++11 -stdlib=libc++
python/bolt/native_wrap.cpp:3053:10: fatal error: 'stdexcept' file not found
#include
^
1 error generated.
error: command 'clang' failed with exit status 1

@dblalock
Copy link
Owner Author

dblalock commented Sep 7, 2018

This might be an issue with the stdlib that you're using. Can you build the following simple program?

main.cpp:

#include <stdexcept>
int main() { return 0; }

with the command:

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libc++

I also just removed the duplicate std= flags from the configuration, which may have caused problems.

@dblalock dblalock reopened this Sep 7, 2018
@goobee
Copy link

goobee commented Sep 7, 2018

i got this by using clang++ ,and g++ is ok

root@sci:/home/xx# clang++ main.cpp -o main.out -std=c++14 -stdlib=libc++ main.cpp:1:9: fatal error: 'stdexcept' file not found #include<stdexcept> ^ 1 error generated. root@sci:/home/xx# root@sci:/home/xx# root@sci:/home/xx# g++ main.cpp

@dblalock
Copy link
Owner Author

dblalock commented Sep 7, 2018

Great. Now what happens when you instead use a different stdlib? I.e.:

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libstdc++

or

$ clang++ main.cpp -o main.out -std=c++14

Also, what do you get when you do:

$ clang -v

@goobee
Copy link

goobee commented Sep 9, 2018

hello,when i follow those two command,

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libstdc++
$ clang++ main.cpp -o main.out -std=c++14

I got the same error

[root@localhost bolt]# clang++ main.cpp -o main.out -std=c++14     -stdlib=libstdc++
In file included from main.cpp:1:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/stdexcept:39:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/string:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/char_traits.h:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/postypes.h:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/cwchar:44:
/usr/include/wchar.h:39:11: fatal error: 'stdarg.h' file not found
# include <stdarg.h>
1 error generated.


[root@localhost bolt]# clang -v
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@dblalock
Copy link
Owner Author

dblalock commented Sep 9, 2018

Hmm. Based on what's here, can you compile a one-line cpp file that just contains:

 #include <cstdarg>

Also, if I'm counting the ..s properly, it appears to be searching in /usr/include/c++/4.8.5/ for all of your headers. This issue suggests that this header is supposed to get added when you install clang. Do you see stdarg.h in that directory? If not, what about when you do:

$ find /usr/include -name 'stdarg.h'

Also, for the sake of others seeing this thread, I edited your post so the error messages appear as code and the "include" doesn't look like a header. Hope that's okay. (The backticks didn't work across the line boundaries.)

@drscotthawley
Copy link

drscotthawley commented Sep 1, 2021

Tried to get this to work on Colab, but got "unterminated string constant" build error from swig after cd bolt && python setup.py install:

swigging cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i to cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp
swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i
cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i:3: Error: Unterminated string constant
error: command 'swig' failed with exit status 1

My minimal Colab example link is here: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing

@mrjzz
Copy link

mrjzz commented Sep 3, 2021

Tried to get this to work on Colab, but got "unterminated string constant" build error from swig after cd bolt && python setup.py install:

swigging cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i to cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp
swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i
cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i:3: Error: Unterminated string constant
error: command 'swig' failed with exit status 1

My minimal Colab example link is here: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing

I encounter this error with my Ubuntu likewise.
It seem to be caused by the upgrade of swig's version up to 4.0 (my swig version in ubuntu is 4.0).But I am not sure.
THere is a similar bug:
InsightSoftwareConsortium/ITK#2026

@husisy
Copy link

husisy commented Sep 4, 2021

I guess the folder cpp/src/external/eigen/.hg is managed by the software Mercurial (something like git) which will lead to the issue Unterminated string constant, so I simply delete this folder to go on the installation.

Then, I struggle with the issue "command clang not exist" on Ubuntu. To solve this, I turn to run this code on Mac.

Also, in setup.py, some items included in the list srcFiles will cause swig to fail, so I add modified several lines in it like below

exclude_file_ext = ['.cxxlist','.cu','.cmake','.in','.css','.entry','.evaluator','.expression','.main','.preamble','.traits']
srcFiles += [x for x in files if not any(x.endswith(y) for y in exclude_file_ext)]

After all above, it seems start to build the project, but fail agian...... The error is like below and I have no idea now...

  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:
  cpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials/PolynomialSolver.h:365:43: error: no member named 'abs' in namespace 'Eigen::numext'; did you mean simply 'abs'?
            if( internal::isMuchSmallerThan(numext::abs(numext::imag(m_roots[i])),
                                            ^~~~~~~~~~~
                                            abs
  cpp/src/external/eigen-trimmed/Eigen/src/Core/GlobalFunctions.h:58:36: note: 'abs' declared here
    EIGEN_ARRAY_DECLARE_GLOBAL_UNARY(abs,scalar_abs_op)
                                     ^
  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:
  cpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials/PolynomialSolver.h:366:43: error: no member named 'abs' in namespace 'Eigen::numext'; did you mean simply 'abs'?
                                            numext::abs(numext::real(m_roots[i])),
                                            ^~~~~~~~~~~
                                            abs
  cpp/src/external/eigen-trimmed/Eigen/src/Core/GlobalFunctions.h:58:36: note: 'abs' declared here
    EIGEN_ARRAY_DECLARE_GLOBAL_UNARY(abs,scalar_abs_op)
                                     ^
  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:

@drscotthawley
Copy link

drscotthawley commented Sep 4, 2021

On Ubuntu, after following @husisy's advice and removing eigen's .hg/ folder, I start getting "unknown file type" errors, such as:

error: unknown file type '.cmake' (from 'cpp/src/external/eigen/CTestConfig.cmake')

^on Colab, and on my Pop_OS (Ubuntu variant) laptop I get:

error: unknown file type '.cu' (from 'cpp/src/external/eigen-trimmed/unsupported/test/cxx11_tensor_argmax_gpu.cu')

EDIT: After modifying setup.py as per @husisy, I'm able to get it down to an error about a missing file that apparently only exists on Apple systems:

clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fdebug-prefix-map=/build/python3.7-LSlbJj/python3.7-3.7.11=. -fstack-protector-strong -Wformat -Werror=format-security -g -fdebug-prefix-map=/build/python3.7-LSlbJj/python3.7-3.7.11=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DNDEBUG=1 -I/usr/local/lib/python3.7/dist-packages/numpy/core/include -Icpp/src/utils -Icpp/src/external -Icpp/src/include -Icpp/src/quantize -Icpp/src/external/eigen -Icpp/src/external/eigen-trimmed -Icpp/src/external/eigen/blas -Icpp/src/external/eigen/unsupported -Icpp/src/external/eigen/debug -Icpp/src/external/eigen/demos -Icpp/src/external/eigen/cmake -Icpp/src/external/eigen/doc -Icpp/src/external/eigen/scripts -Icpp/src/external/eigen/bench -Icpp/src/external/eigen/Eigen -Icpp/src/external/eigen/failtest -Icpp/src/external/eigen/blas/fortran -Icpp/src/external/eigen/unsupported/Eigen -Icpp/src/external/eigen/unsupported/Eigen/src -Icpp/src/external/eigen/unsupported/Eigen/CXX11 -Icpp/src/external/eigen/unsupported/Eigen/src/SparseExtra -Icpp/src/external/eigen/unsupported/Eigen/src/Splines -Icpp/src/external/eigen/unsupported/Eigen/src/NumericalDiff -Icpp/src/external/eigen/unsupported/Eigen/src/AutoDiff -Icpp/src/external/eigen/unsupported/Eigen/src/NonLinearOptimization -Icpp/src/external/eigen/unsupported/Eigen/src/Skyline -Icpp/src/external/eigen/unsupported/Eigen/src/IterativeSolvers -Icpp/src/external/eigen/unsupported/Eigen/src/KroneckerProduct -Icpp/src/external/eigen/unsupported/Eigen/src/LevenbergMarquardt -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions -Icpp/src/external/eigen/unsupported/Eigen/src/MatrixFunctions -Icpp/src/external/eigen/unsupported/Eigen/src/FFT -Icpp/src/external/eigen/unsupported/Eigen/src/Polynomials -Icpp/src/external/eigen/unsupported/Eigen/src/BVH -Icpp/src/external/eigen/unsupported/Eigen/src/MoreVectorization -Icpp/src/external/eigen/unsupported/Eigen/src/Eigenvalues -Icpp/src/external/eigen/unsupported/Eigen/src/EulerAngles -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions/arch -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions/arch/CUDA -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/Tensor -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/ThreadPool -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/TensorSymmetry -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/util -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/TensorSymmetry/util -Icpp/src/external/eigen/debug/msvc -Icpp/src/external/eigen/debug/gdb -Icpp/src/external/eigen/demos/mix_eigen_and_c -Icpp/src/external/eigen/demos/opengl -Icpp/src/external/eigen/demos/mandelbrot -Icpp/src/external/eigen/doc/snippets -Icpp/src/external/eigen/doc/special_examples -Icpp/src/external/eigen/doc/examples -Icpp/src/external/eigen/bench/spbench -Icpp/src/external/eigen/bench/btl -Icpp/src/external/eigen/bench/tensors -Icpp/src/external/eigen/bench/perf_monitoring -Icpp/src/external/eigen/bench/btl/generic_bench -Icpp/src/external/eigen/bench/btl/cmake -Icpp/src/external/eigen/bench/btl/libs -Icpp/src/external/eigen/bench/btl/data -Icpp/src/external/eigen/bench/btl/actions -Icpp/src/external/eigen/bench/btl/generic_bench/timers -Icpp/src/external/eigen/bench/btl/generic_bench/utils -Icpp/src/external/eigen/bench/btl/generic_bench/init -Icpp/src/external/eigen/bench/btl/generic_bench/static -Icpp/src/external/eigen/bench/btl/libs/ublas -Icpp/src/external/eigen/bench/btl/libs/blaze -Icpp/src/external/eigen/bench/btl/libs/mtl4 -Icpp/src/external/eigen/bench/btl/libs/tensors -Icpp/src/external/eigen/bench/btl/libs/BLAS -Icpp/src/external/eigen/bench/btl/libs/blitz -Icpp/src/external/eigen/bench/btl/libs/eigen2 -Icpp/src/external/eigen/bench/btl/libs/eigen3 -Icpp/src/external/eigen/bench/btl/libs/STL -Icpp/src/external/eigen/bench/btl/libs/tvmet -Icpp/src/external/eigen/bench/btl/libs/gmm -Icpp/src/external/eigen/bench/perf_monitoring/gemm -Icpp/src/external/eigen/Eigen/src -Icpp/src/external/eigen/Eigen/src/QR -Icpp/src/external/eigen/Eigen/src/SVD -Icpp/src/external/eigen/Eigen/src/CholmodSupport -Icpp/src/external/eigen/Eigen/src/OrderingMethods -Icpp/src/external/eigen/Eigen/src/PaStiXSupport -Icpp/src/external/eigen/Eigen/src/LU -Icpp/src/external/eigen/Eigen/src/IterativeLinearSolvers -Icpp/src/external/eigen/Eigen/src/plugins -Icpp/src/external/eigen/Eigen/src/Jacobi -Icpp/src/external/eigen/Eigen/src/Core -Icpp/src/external/eigen/Eigen/src/Householder -Icpp/src/external/eigen/Eigen/src/PardisoSupport -Icpp/src/external/eigen/Eigen/src/SparseQR -Icpp/src/external/eigen/Eigen/src/misc -Icpp/src/external/eigen/Eigen/src/MetisSupport -Icpp/src/external/eigen/Eigen/src/UmfPackSupport -Icpp/src/external/eigen/Eigen/src/SuperLUSupport -Icpp/src/external/eigen/Eigen/src/Cholesky -Icpp/src/external/eigen/Eigen/src/SparseCholesky -Icpp/src/external/eigen/Eigen/src/StlSupport -Icpp/src/external/eigen/Eigen/src/SPQRSupport -Icpp/src/external/eigen/Eigen/src/Eigenvalues -Icpp/src/external/eigen/Eigen/src/Geometry -Icpp/src/external/eigen/Eigen/src/SparseCore -Icpp/src/external/eigen/Eigen/src/SparseLU -Icpp/src/external/eigen/Eigen/src/LU/arch -Icpp/src/external/eigen/Eigen/src/Core/arch -Icpp/src/external/eigen/Eigen/src/Core/products -Icpp/src/external/eigen/Eigen/src/Core/functors -Icpp/src/external/eigen/Eigen/src/Core/util -Icpp/src/external/eigen/Eigen/src/Core/arch/SSE -Icpp/src/external/eigen/Eigen/src/Core/arch/AltiVec -Icpp/src/external/eigen/Eigen/src/Core/arch/CUDA -Icpp/src/external/eigen/Eigen/src/Core/arch/AVX512 -Icpp/src/external/eigen/Eigen/src/Core/arch/AVX -Icpp/src/external/eigen/Eigen/src/Core/arch/Default -Icpp/src/external/eigen/Eigen/src/Core/arch/NEON -Icpp/src/external/eigen/Eigen/src/Core/arch/ZVector -Icpp/src/external/eigen/Eigen/src/Geometry/arch -Icpp/src/external/eigen-trimmed/unsupported -Icpp/src/external/eigen-trimmed/Eigen -Icpp/src/external/eigen-trimmed/unsupported/test -Icpp/src/external/eigen-trimmed/unsupported/doc -Icpp/src/external/eigen-trimmed/unsupported/bench -Icpp/src/external/eigen-trimmed/unsupported/Eigen -Icpp/src/external/eigen-trimmed/unsupported/test/mpreal -Icpp/src/external/eigen-trimmed/unsupported/doc/snippets -Icpp/src/external/eigen-trimmed/unsupported/doc/examples -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11 -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SparseExtra -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Splines -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/NumericalDiff -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/AutoDiff -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/NonLinearOptimization -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Skyline -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/IterativeSolvers -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/KroneckerProduct -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/LevenbergMarquardt -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/MatrixFunctions -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/FFT -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/BVH -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/MoreVectorization -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Eigenvalues -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/EulerAngles -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions/arch -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions/arch/GPU -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/Tensor -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/ThreadPool -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/TensorSymmetry -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/util -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/TensorSymmetry/util -Icpp/src/external/eigen-trimmed/Eigen/src -Icpp/src/external/eigen-trimmed/Eigen/src/QR -Icpp/src/external/eigen-trimmed/Eigen/src/SVD -Icpp/src/external/eigen-trimmed/Eigen/src/CholmodSupport -Icpp/src/external/eigen-trimmed/Eigen/src/OrderingMethods -Icpp/src/external/eigen-trimmed/Eigen/src/PaStiXSupport -Icpp/src/external/eigen-trimmed/Eigen/src/LU -Icpp/src/external/eigen-trimmed/Eigen/src/IterativeLinearSolvers -Icpp/src/external/eigen-trimmed/Eigen/src/plugins -Icpp/src/external/eigen-trimmed/Eigen/src/Jacobi -Icpp/src/external/eigen-trimmed/Eigen/src/Core -Icpp/src/external/eigen-trimmed/Eigen/src/Householder -Icpp/src/external/eigen-trimmed/Eigen/src/PardisoSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SparseQR -Icpp/src/external/eigen-trimmed/Eigen/src/misc -Icpp/src/external/eigen-trimmed/Eigen/src/MetisSupport -Icpp/src/external/eigen-trimmed/Eigen/src/UmfPackSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SuperLUSupport -Icpp/src/external/eigen-trimmed/Eigen/src/Cholesky -Icpp/src/external/eigen-trimmed/Eigen/src/SparseCholesky -Icpp/src/external/eigen-trimmed/Eigen/src/StlSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SPQRSupport -Icpp/src/external/eigen-trimmed/Eigen/src/Eigenvalues -Icpp/src/external/eigen-trimmed/Eigen/src/Geometry -Icpp/src/external/eigen-trimmed/Eigen/src/SparseCore -Icpp/src/external/eigen-trimmed/Eigen/src/SparseLU -Icpp/src/external/eigen-trimmed/Eigen/src/LU/arch -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch -Icpp/src/external/eigen-trimmed/Eigen/src/Core/products -Icpp/src/external/eigen-trimmed/Eigen/src/Core/functors -Icpp/src/external/eigen-trimmed/Eigen/src/Core/util -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/SSE -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/AltiVec -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/CUDA -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/AVX -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/Default -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/NEON -Icpp/src/external/eigen-trimmed/Eigen/src/Geometry/arch -I/content/bolt/python/bolt -I/usr/include/python3.7m -c python/bolt/native_wrap.cpp -o build/temp.linux-x86_64-3.7/python/bolt/native_wrap.o -std=c++14 -fno-rtti -march=native -ffast-math -stdlib=libc++
python/bolt/native_wrap.cpp:3053:10: fatal error: 'stdexcept' file not found
#include <stdexcept>
         ^~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1

Colab link is the same as before: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing#scrollTo=ks4orqb0Sxrs

@drscotthawley
Copy link

drscotthawley commented Sep 7, 2021

Update: Above Colab goes significantly further than before but still fails.
'Fixed' the above Apple-specific error after reading further up this Issue (woops) by creating a blank file. Then spent about 7 hours on Saturday. Seems that clang is getting called where clang++ should be, and that bolt code has hard-coded "eigen/Core" where it should read (and other libraries use) "Eigen/Core" or "eigen3/Eigen/Core". Is this an artifact using a case-unspecific filesystem like Mac?

In file included from python/bolt/native_wrap.cpp:3129:
In file included from /content/bolt/python/bolt/../../cpp/src/include/public.hpp:19:
In file included from cpp/src/external/eigen/Eigen/Core:96:
In file included from /usr/include/c++/7.5.0/complex:45:
In file included from /usr/include/c++/7.5.0/sstream:38:
In file included from /usr/include/c++/7.5.0/istream:38:
In file included from /usr/include/c++/7.5.0/ios:42:
In file included from /usr/include/c++/7.5.0/bits/ios_base.h:46:
/usr/include/c++/7.5.0/system_error:94:13: error: unknown type name '__cow_string'
    virtual __cow_string
            ^
/usr/include/c++/7.5.0/system_error:341:36: error: expected class name
  class system_error : public std::runtime_error
                                   ^
/usr/include/c++/7.5.0/system_error:348:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__ec.message()), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:351:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + ": " + __ec.message()), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:354:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + (": " + __ec.message())), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:360:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(error_code(__v, __ecat).message()),
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:364:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + ": " + error_code(__v, __ecat).message()),
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
python/bolt/native_wrap.cpp:3141:12: fatal error: 'eigen/Core' file not found
  #include "eigen/Core"
           ^~~~~~~~~~~~
8 errors generated.

@dblalock Let us know when we can try this out! Looking forward to it. Until then, 👋

@362502anct
Copy link

Hello, when I install the bolt on my macbook, I follow the installing process but meet the following problem.
swigging cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i to cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in_wrap.cpp swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i:1: Error: Illegal token '?'. cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i:1: Error: Syntax error in input(1). error: command 'swig' failed with exit status 1
Even changing the version of swig can not solve this problem. So I ask what can I do? Thanks.

@dblalock
Copy link
Owner Author

dblalock commented Sep 8, 2021

Thanks for everyone's feedback and work on this (special thanks to @drscotthawley, as well as @husisy and @calvinmccarter, who've put a bunch of time into helping to get all this working). I'm looking into this, but it's kind of slow going at the moment (especially since I'm doing about 70h/week at my day job and don't have much time left over). I appreciate everyone's enthusiasm and patience.

@calvinmccarter
Copy link
Contributor

@dblalock With a bit more documentation around the C++ API, I'd be happy to write a basic command-line interface. Obviously getting and returning matrices via file i/o is slow, but it would be sufficient for me.

@mneilly
Copy link
Contributor

mneilly commented Oct 12, 2021

FWIW - I have a branch that has changes so that things can be built and run on Linux and Mac. The changes basically add a CMakeLists.txt to build with cmake, add a kmc2 submodule that works with python3 and setup.py changes so the python package builds. There is a build.sh that was tested to work on Debian 11 and Mac OS X 10.14.6.

https://github.com/mneilly/bolt/tree/mneilly/fixes

@cTatu
Copy link

cTatu commented Oct 13, 2021

Now it works for me! Good job thank you
The only error I get now is when I use accuracy=high. With accuracy low and medium everything works fine.
this was my test

/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py:104: RuntimeWarning: invalid value encountered in true_divide
  seeds = kmc2.kmc2(X, k).astype(np.float32)
Traceback (most recent call last):
  File "tst.py", line 14, in <module>
    enc.fit(X)
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 464, in fit
    centroids = _learn_centroids(X, ncentroids=ncentroids,
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 140, in _learn_centroids
    centroids, labels = kmeans(X_in, ncentroids)
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 104, in kmeans
    seeds = kmc2.kmc2(X, k).astype(np.float32)
  File "kmc2.pyx", line 97, in kmc2.kmc2 (kmc2.c:2646)
  File "mtrand.pyx", line 928, in numpy.random.mtrand.RandomState.choice
ValueError: probabilities contain NaN

@dblalock
Copy link
Owner Author

dblalock commented Jan 2, 2022

@mneilly this is awesome, thanks! I'll be happy to merge it if you open a PR.

@charlswzc
Copy link

@dblalock @mneilly If anybody runs BOLT perfectly, can you you guys share the information of your machine? Like OS version, swig version, and so on. Many thanks!

I tried win10, ubuntu 18.04/20.01, still not working. [sad]

@mneilly
Copy link
Contributor

mneilly commented Apr 29, 2022

@dblalock I opened the PR.
@charlswzc The PR has a Dockerfile which works that you can use as a reference.

@windspirit95
Copy link

windspirit95 commented Jun 23, 2022

@dblalock @mneilly If anybody runs BOLT perfectly, can you you guys share the information of your machine? Like OS version, swig version, and so on. Many thanks!

I tried win10, ubuntu 18.04/20.01, still not working. [sad]

Finally, I could install Bolt perfectly, with kmc2 install

  • Ubuntu version: 18.04
  • swig 3.0.12
  • Install Clang - Ubuntu
sudo apt install clang
  • Install Bolt:
git clone -b mneilly/fixes https://github.com/mneilly/bolt.git
cd bolt
python setup.py install
cd ../
  • Install kmc2
git clone -b mneilly/cythonize https://github.com/mneilly/kmc2.git
cd kmc2
python setup.py install
cd ../

Thanks to @mneilly for the PR ^^

@dblalock
Copy link
Owner Author

dblalock commented Jun 24, 2022

Yes, big thanks to mneilly! For any future readers, I actually merged this branch into Bolt, so in theory you should get the same results in the main repo now.

That said, I would never bet on the robustness of mostly-unmaintained code spanning two languages with legacy FFI interfaces and x86-specific libraries, so worth giving mneilly's branch a try if this repo doesn't work. The docker container he wrote (see BUILD.md should be more reliable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests