Help installing on Win64 with MSYS2 (mingw64 compilers)

[kovalp]

Describe the issue

I am trying to get sisl installed on Windows, in plain Python installation (i.e. not Anaconda).
My best take is to install MSYS2, install gcc and gfortran packages there and then pip gets farther as seen in the output below

koval@DESKTOP-B5CLTB9 MINGW64 ~/programs
$ pip install sisl
Collecting sisl
  Downloading sisl-0.10.0.tar.gz (3.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied: numpy>=1.13 in c:\users\koval\appdata\local\programs\python\python37\lib\site-packages (from sisl) (1.18.1)
Requirement already satisfied: pyparsing>=1.5.7 in c:\users\koval\appdata\local\programs\python\python37\lib\site-packages (from sisl) (2.4.6)
Collecting netCDF4
  Downloading netCDF4-1.5.4.tar.gz (793 kB)
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\koval\appdata\local\programs\python\python37\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\koval\\AppData\\Local\\Temp\\pip-install-v801fhjq\\netCDF4\\setup.py'"'"'; __file__='"'"'C:\\Users\\koval\\AppData\\Local\\Temp\\pip-install-v801fhjq\\netCDF4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\koval\AppData\Local\Temp\pip-pip-egg-info-tyxwvna8'
         cwd: C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\
    Complete output (16 lines):
    reading from setup.cfg...

        HDF5_DIR environment variable not set, checking some standard locations ..
    checking C:\Users\koval ...
    checking /usr/local ...
    checking /sw ...
    checking /opt ...
    checking /opt/local ...
    checking /usr ...
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\setup.py", line 397, in <module>
        _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
      File "C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\setup.py", line 345, in _populate_hdf5_info
        raise ValueError('did not find HDF5 headers')
    ValueError: did not find HDF5 headers
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Version details
Run the below code and add to issue (if an issue is relevant for the issue):

import sys
print(sys.version)
import sisl
print(sisl.__version__)
print(sisl.__git_revision__)

$ ipython
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys
...: print(sys.version)
3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)]

import sisl is not working yet.

[kovalp]

If I am installing using setup.py, then I get a different problem.

sisl-install: using internal shipped revisions
running install
running bdist_egg
running egg_info
running build_src
build_src
building extension "sisl.io.siesta._siesta" sources
f2py options: []
  adding 'build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta\fortranobject.c' to sources.
  adding 'build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta' to include_dirs.
  adding 'build\src.win-amd64-3.7\sisl\io\siesta\_siesta-f2pywrappers2.f90' to sources.
building extension "sisl._indices" sources
building extension "sisl._math_small" sources
building extension "sisl.physics._bloch" sources
building extension "sisl.physics._matrix_ddk" sources
building extension "sisl.physics._matrix_dk" sources
building extension "sisl.physics._matrix_k" sources
building extension "sisl.physics._matrix_phase3" sources
building extension "sisl.physics._matrix_phase3_nc" sources
building extension "sisl.physics._matrix_phase3_so" sources
building extension "sisl.physics._matrix_phase_nc_diag" sources
building extension "sisl.physics._matrix_phase_nc" sources
building extension "sisl.physics._matrix_phase" sources
building extension "sisl.physics._matrix_phase_so" sources
building extension "sisl.physics._phase" sources
building extension "sisl._sparse" sources
building extension "sisl._supercell" sources
build_src: building npy-pkg config files
writing sisl.egg-info\PKG-INFO
writing dependency_links to sisl.egg-info\dependency_links.txt
writing entry points to sisl.egg-info\entry_points.txt
writing requirements to sisl.egg-info\requires.txt
writing top-level names to sisl.egg-info\top_level.txt
reading manifest file 'sisl.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'doc'
warning: no files found matching '*.md' under directory 'toolbox'
warning: no previously-included files found matching 'DEPLOY.md'
warning: no previously-included files matching '*.html' found anywhere in distribution
warning: no previously-included files matching '*.sh' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'benchmarks'
no previously-included directories found matching 'deprecated'
no previously-included directories found matching 'developments'
no previously-included directories found matching 'ci'
no previously-included directories found matching 'files'
no previously-included directories found matching '__pycache__'
writing manifest file 'sisl.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
copying sisl\info.py -> build\lib.win-amd64-3.7\sisl
running build_ext
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using EnsureBuildExt
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Found executable C:\Program Files\Git\usr\bin\DF.exe
Could not locate executable C:\Program
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Found executable C:\msys64\mingw64\bin\gfortran.exe
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.1.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --disable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev3, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (Rev3, Built by MSYS2 project) 
customize Gnu95FCompiler
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/lto-wrapper.exe
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.1.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --disable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev3, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (Rev3, Built by MSYS2 project) 
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
customize Gnu95FCompiler using EnsureBuildExt
building 'sisl.io.siesta._siesta' extension
compiling C sources
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/

[zerothi]

Hmm...

Your first post was because the HDF5 library + headers were not available.

Your second post is because the gfortran compilation fails with some weird looking flags. I don't know how they entered, i.e. I can't find them in numpy which is used for the extensions.

I don't have a windows box to test this with. So I can't be of much help. I guess the fortran problem could be googleable?

[kovalp]

Dear Nick,

Thank you for your comments.

After installing pip install h5py and the microsoft build tools in the following configuration

the installation with pip install sisl finished reporting success.

However, I still could not import sisl module

koval@DESKTOP-B5CLTB9 MINGW64 ~
$ python -c "import sisl; print(sisl.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\__init__.py", line 128, in <module>
    from . import io
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\__init__.py", line 284, in <module>
    from .tbtrans import *
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\tbtrans\__init__.py", line 71, in <module>
    from .delta import *
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\tbtrans\delta.py", line 16, in <module>
    from ..siesta._siesta import siesta_sc_off
ImportError: DLL load failed: The specified module could not be found.
DLL load failed: The specified module could not be found.

Any hint is appreciated.
I suspect the failing import is a Fortran extension (..siesta._siesta), isn't?

PS. I am using a free unregistered windows in the virtualbox to do these trials.

PPS. The Fortran compiler has been provided by MSYS2 tools. The C compiler has been provided by Visual Studio. It would be nice to convince pip install sisl to use C compiler from MSYS2.

[zerothi]

Hmmm, could you try:

pip install -vvv --global-option "build --compiler=<compiler> --fcompiler=<compiler>" sisl

where <compiler> is one of msvc9, msvc, _msvc, cygwinc.
The -vvv should give additional details on the compilation process. However, it is quite weird it successfully installs but fails to install the fortran-sources.

[zerothi]

Oh, --fcompiler=gnu should also be tried!

[kovalp]

Thank you for your comments. I had tried some of these options. I guess a minimal sisl with just one small subroutine using a fortran function would help in debugging. Whenever I use --global-option, pip recompiles cython and numpy. This takes a load of time and produces the non-working installation. I used -v. It uncovers a lot indeed, but I am lacking a better control on the compilers and on what has to be compiled. I had more control when using python setup.py install, but still no luck (because my msys2 gfortran accepts posix slashes in paths, not windows ones I think).

[zerothi]

Yeah, pip is not always easy to work with.

To better control the environment with pip you could do something like this (this command will require you have all dependencies pre-installed and reachable in PYTHONPATH):

pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=cygwinc --global-option --fcompiler=gnu95 sisl

What does this give of output?

[kovalp]

$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=cygwinc --global-option --fcompiler=gnu95 sisl
Defaulting to user installation because normal site-packages is not writeable
Ignoring indexes: https://pypi.org/simple
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-ephem-wheel-cache-t0xsaw0a
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Initialized build tracking at C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Created build tracker: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Entered build tracker: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-install-3q197tre
0 location(s) to search for versions of sisl:
Given no hashes to check 0 links for project 'sisl': discarding no candidates
C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\commands\install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
ERROR: Could not find a version that satisfies the requirement sisl (from versions: none)
ERROR: No matching distribution found for sisl
Exception information:
Traceback (most recent call last):
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\cli\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\cli\req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\commands\install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 313, in _get_abstract_dist_for
    self._populate_link(req)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 279, in _populate_link
    req.link = self.finder.find_requirement(req, upgrade)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\index\package_finder.py", line 930, in find_requirement
    req)
pip._internal.exceptions.DistributionNotFound: No matching distribution found for sisl
Removed build tracker: 'C:\\Users\\kovalp\\AppData\\Local\\Temp\\pip-req-tracker-d_svvy0z'

[zerothi]

Ah, yes, download the sisl.tar.gz, extract, cd, and do pip .... . (i.e. replace sisl with .)

[kovalp]

pip.out.txt

I know how to solve that. I will solve. However, the problem is that it starts to ask about visual studio C/C++.

cygwinc I don't have.

[zerothi]

Hmm, I don't know what this means, but this seems like a general case.. I didn't get if you know what to do, or if you still need help ;)

It seems this is the correct direction?

[kovalp]

This particular trouble I can solve (by installing Visual C). I will see what comes later. I still use your experience on this.

[kovalp]

Below is the output. The installation is successful.
pip.out.txt

However, I still experience the same trouble when importing sisl

kovalp@DESKTOP-AT1KGE5 MINGW64 ~/Downloads/sisl-master/sisl-master
$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=gcc --global-option --fcompiler=gnu95 . &>pip.out

kovalp@DESKTOP-AT1KGE5 MINGW64 ~/Downloads/sisl-master/sisl-master
$ cd

kovalp@DESKTOP-AT1KGE5 MINGW64 ~
$ python -c "import sisl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\__init__.py", line 128, in <module>
    from . import io
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\__init__.py", line 284, in <module>
    from .tbtrans import *
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\tbtrans\__init__.py", line 71, in <module>
    from .delta import *
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\tbtrans\delta.py", line 16, in <module>
    from ..siesta._siesta import siesta_sc_off
ImportError: DLL load failed: No se puede encontrar el m�dulo especificado.
DLL load failed: No se puede encontrar el m�dulo especificado.

[zerothi]

Hmm, I must admit I have no clue...
It seems that the fortran code compiled does not work correctly.

It is also unclear to me what the importance of the compilers are. You seem to be using msys64 for fortran sources, but MSVC for c-code, so basically you are compiling with one compiler and linking with another. This could be the reason.

Could you try with

$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=mingw32 --global-option --fcompiler=gnu95 . &>pip.out

Or try something that changes the C-compiler. Play around with --compiler options I think could be a test, check in the output how it does the final linking.
In your output, this line is the key:

  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\msys64\mingw64\lib\gcc\x86_64-w64-mingw32\9.1.0 /LIBPATH:c:\program files\python37\libs /LIBPATH:c:\program files\python37\PCbuild\amd64 /LIBPATH:c:\program files\python37\libs /LIBPATH:c:\program files\python37\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /EXPORT:PyInit__siesta build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\sisl\io\siesta\_siestamodule.obj build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta\fortranobject.obj build\temp.win-amd64-3.7\Release\libio_m.DL5GA7BPLQZXMPFPWEFJ2SIMXZB6UBNR.gfortran-win_amd64.lib /OUT:build\lib.win-amd64-3.7\sisl\io\siesta\_siesta.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\sisl\io\siesta\_siesta.cp37-win_amd64.lib

I.e. the module _siesta.cp...

[kovalp]

Thank you for your comments. The problem is that it is difficult to control the C compiler and there is no Fortran compiler provided by the visual studio.