virtualenv 1.11.6 not working with MSYS2

[pe224]

I'm using the latest version of MSYS2 with Python 2.7.8 as the only system-wide Python.
Pip installed virtualenv 1.11.6 without problems.
Executing virtualenv myenv results in

Traceback (most recent call last):
  File "C:\msys64\home\eib/myvenv/lib/python2.7/site.py", line 703, in <module>
    main()
  File "C:\msys64\home\eib/myvenv/lib/python2.7/site.py", line 692, in main
    aliasmbcs()
  File "C:\msys64\home\eib/myvenv/lib/python2.7/site.py", line 515, in aliasmbcs
    import locale, codecs
  File "C:/msys64/home/eib/myvenv/lib/python2.7/locale.py", line 19, in <module>
    import functools
ImportError: No module named functools
New python executable in myvenv/Scripts/python.exe
ERROR: The executable myvenv/Scripts/python.exe is not functioning
ERROR: It thinks sys.prefix is u'c:\\msys64\\home\\eib' (should be u'c:\\msys64\\home\\eib\\myvenv')
ERROR: virtualenv is not compatible with this system or executable
Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

Supplying additional arguments like an explicit --python didn't change anything.
Now there seem to be two problems:
First, the virtualenv is not created completely. Parts of myenv/lib are missing and there are no activation scripts.
Second, there is only a non-functioning python in myenv/Scripts/python.exe

Since there is no Python dll, I tried copying around libpython2.7.dll.a and libpython2.7.a, but to no avail. What made the myenv/Scripts/python.exe to work was to symlink the working Python lib/python2.7 to myvenv/lib/python2.7, then it could find the standard library.

Why does the virtualenv creation process stop in the middle? It looks like everything could be brought to work. Also, it seems that whole include/lib folders are copied, although MSYS2 is configured to do native windows symlinks with "ln -s". On my mac, virtualenv creates correct symlinks.

Is it a path issue? $MSYS_HOME is correctly set to /c/msys64
What's going on? Thanks for any pointers.

[pfmoore]

To clarify, are you saying that the system Python is built using msys? Or is it a standard python.org build? If the latter, in what sense is msys relevant here?

If the Python interpreter is built with msys, then we don't explicitly support that environment. It's effectively a sort of stripped down cygwin environment, but it's not really cygwin. Let's clarify precisely what the environment you're working in is, before jumping to any conclusions, though...

[pe224]

The system Python is built with MinGW64. Is this supported? Could you give hints where to look/what fails - for some kind of hack?

You're right that MSYS doesn't matter in that case, as I can call the virtualenv command from a Windows terminal and still get the same error. I just thought that now that there is a kind of symlink from win7 on (and it is called in the standard way via ln in MSYS), it would be nice not having to copy the whole Python over - as it is e.g. nicely done by virtualenv on Mac

[pfmoore]

It sounds to me like the Python build is not identifying sys.prefix correctly, which implies that it's an issue with your Python build rather than a virtualenv issue, and as it's a custom build, it's hard to see how to investigate this further.

If you can clarify precisely how the system Python was built, I can see if I can reproduce the build here and investigate further, but that's about all I can think of.

[pe224]

Thanks for your offer. I think we can get this to work without going back to the build. However, I know not enough about virtualenv internals, so maybe you can support me a bit.
So far, I made following changes to virtualenv.py

$ diff .../site-packages/virtualenv_original.py .../site-packages/virtualenv.py
52a53
> is_msys = (sys.platform == 'win32' and 'MSYS' in os.environ)
293a295,296
> if is_msys:
>     REQUIRED_MODULES.extend(['functools'])
472c475
<     if symlink and hasattr(os, 'symlink') and not is_win:
---
>     if symlink and hasattr(os, 'symlink'):
1027a1031,1034
>         if is_msys:
>             lib_dir = join(home_dir, 'lib', py_version)
>             inc_dir = join(home_dir, 'include')
>             bin_dir = join(home_dir, 'bin')

is_msys: detects whether MSYS is present by an environment variable.
Then, we need to make sure, we have functools, otherwise we cannot call locale.py later (for some encoding)
I added a custom os.symlink using windows API, so we can remove the (anyways redundant) is_win when asking for os.symlink and use symlinks in Windows from now on if they're available.
Finally, MSYS doesn't like the default Python folder layout, so I changed it to the non-windows default.

Now, running a virtualenv --no-pip --no-setuptools myenv runs perfectly and generates valid activation scripts and everything.

However, when the script tries to install pip and setuptools, following error occurs:

New python executable in bla/bin/python.exe
Installing setuptools, pip...
  Complete output from command C:/msys64/home/eib/bla/bin/python.exe -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:/msys64/mingw64/lib/python2.7/site-packages/virtualenv_support/pip-1.5.6-py2.py3-none-any.whl/pip/__init__.py", line 3, in <module>
ImportError: No module named optparse
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:/msys64/mingw64/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:/msys64/mingw64/bin/virtualenv.exe/__main__.py", line 9, in <module>
  File "C:/msys64/mingw64/lib/python2.7/site-packages/virtualenv.py", line 827, in main
    symlink=options.symlink)
  File "C:/msys64/mingw64/lib/python2.7/site-packages/virtualenv.py", line 995, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "C:/msys64/mingw64/lib/python2.7/site-packages/virtualenv.py", line 963, in install_wheel
    'PIP_NO_INDEX': '1'
  File "C:/msys64/mingw64/lib/python2.7/site-packages/virtualenv.py", line 905, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command C:/msys64/home/eib/bla/bin/python.exe -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

So somehow, it cannot find some standard library modules...
At first, I thought "who cares, I just install pip by myself. Could hook it to post activation anyways". But doing a myenv/bin/python.exe get-pip.py shows also lack of standard library features: ImportError: No module named base64
This is where I stop understanding. Ok, I could manually import all those modules needed like I did with functools, but that can't be true. Why has the resulting myenv/bin/python.exe not acccess to the whole standard library? How does this normally work?
Thanks for helping.

[pfmoore]

I'll have a proper look this evening, but my problem here is that if it's a problem with how the Python executable is built, I'd like to know how "official" that Python build is before we bake a workaround into the virtualenv code. If it's something you've built yourself, my instinct is to say you should fix your build. If it's something provided by msys, I'd like to know what usage they support (typically msys tools are not supported for use outside of the msys environment, and the pathname format in your tracebacks makes me think that's not what you're doing.

So where precisely did you get your build of Python from?

[pe224]

Yes. So I'm going to answer this now :)
MSYS2 ported the arch-linux package manager Pacman. It has access to these repositories. There are msys2/packages and mingw-w64/packages - both provide a Python build.
However, since I want to combine Python with mingw-w64-gcc anyways AFAIK only the ming64 package is reasonable (because of automatic handling of dependencies by Pacman, i.e. MSYS2 package with Python2 dependency doesn't accept an installed MinGW64-Python2 and wants to install MSYS2-Python2. This means it's either one or the other).
Inside the ming64 packages you find a mingw-w64-x86_64-python2-2.7.8-5-any.pkg.tar.xz. This is what I used. So it's the "official" MSYS2-MinGW64-Python2 build.
From the description of MSYS2

The set of software-building recipes (PKGBUILDs) for MSYS2 itself are at:
https://github.com/Alexpux/MSYS2-packages
.. and those for MinGW-w64 (i.e. native Windows software) are at:
https://github.com/Alexpux/MINGW-packages

I think you could get the build options for the Python in question from https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2

For the problem at hand:

$ myenv/bin/python.exe -c "import sys; print sys.path"
['',
'C:/msys64/home/eib/bla/lib/python27.zip',
'C:/msys64/home/eib/bla/lib/python2.7',
'C:/msys64/home/eib/bla/lib/python2.7/plat-win32',
'C:/msys64/home/eib/bla/lib/python2.7/lib-tk',
'C:/msys64/home/eib/bla/lib/python2.7/lib-old',
'C:/msys64/home/eib/bla/lib/python2.7/lib-dynload',
'C:/msys64/mingw64',
'C:/msys64/mingw64/Lib',
'C:/msys64/mingw64/DLLs',
'C:/msys64/home/eib/bla/lib/python2.7/site-packages']

doesn't contain C:/msys64/mingw64/lib/python2.7 as last entry. So it cannot find the native Python standard library as fallback.
However, I can't see where in the script you define the sys.path of the new virtualenv executable.

Edit:
I see now. You write a custom site.py, which sets the sys.path. There is the statement elif sys.platform == 'win32': which puts stuff like Lib and DLLs into path.
The main problem is, that this MSYS stuff is basically on a Windows platform, but has a Python folder structure like on Unix. Bad hacks incoming...

So, with the patch below, I got everything to work. I found it simplest to treat MSYS as is_win = False. This way, you have the correct folder structure. Then, I put a custom sitecustomize.py into the virtualenv/lib/python2.7 which appends the native standard library to the sys.path.
Also, detection via environment variable only works when virtualenv.exe is called from MSYS terminal. But you could call it from cmd.exe Windows shell as well. So the detection method should be refined.

Edit2: Changed the detection to whether 'mingw' is in the path to the python executable. A bit better...

$ diff virtualenv_original.py virtualenv.py
50c50
< is_win = (sys.platform == 'win32')
---
> is_win = (sys.platform == 'win32' and not 'mingw' in os.__file__.lower())
52a53
> is_msys = (sys.platform == 'win32' and 'mingw' in os.__file__.lower())
293a295,296
> if is_msys:
>     REQUIRED_MODULES.extend(['functools'])
1263c1266
<     if is_win and os.path.exists(os.path.join(pcbuild_dir, 'build.bat')):
---
>     if (is_win or is_msys) and os.path.exists(os.path.join(pcbuild_dir, 'build.bat')):
1391a1395,1396
>         if is_msys:
>             required_symlinks = [elt + '.exe' for elt in required_symlinks]
1408c1413
<     if is_win and ' ' in py_executable:
---
>     if (is_win or is_msys) and ' ' in py_executable:
1465a1471,1476
>     if is_msys:
>         site_customize = os.path.join(lib_dir, 'site-packages', 'sitecustomize.py')
>         content = ("import sys\n"
>                    "sys.path.append(\"%s\")" % os.path.dirname(os.__file__))
>         writefile(site_customize, content)
>
1471c1482
<     if is_win or is_jython and os._name == 'nt':
---
>     if is_win or is_msys or is_jython and os._name == 'nt':

[pfmoore]

OK. Thanks for the explanation, I think I see what is going on now. MSYS2 seems to be slightly different from MSYS (which is what I have experience with) in that MSYS is purely intended as a build environment for mingw-based programs, whereas MSYS2 seems to be intended to be more usable as a user environment (which explains why it has Python - that's very much a user tool rather than a build tool).

On that basis, MSYS2 python should really be cygwin-based (as the site says, "MSYS2 is a Cygwin-derived Posix-like environment for Windows"). The cygwin builds of Python use a much more Unix-like file layout, which might be better for you :-) Also, virtualenv supports cygwin Python, so the code for that should be a much better fit for what you want.

Basically, my view is that the MSYS2 python should be declaring itself as a cygwin-style build. That will likely work much better with virtualenv (and probably other packages as well). I do not think virtualenv should be supporting a third strain of Windows Python build - if the neither standard (win32) nor the cygwin flavours work for you, you (or the MSYS2 project) will probably need to maintain your own patched version of virtualenv.

Some specific points on your suggestions:

I'm not keen on is_msys being keyed off an environment variable existing. What if I set %MSYS% manually, I certainly wouldn't expect virtualenv to stop working. You need to find a more reliable way of detecting MSYS2.

Although symlinks are available on Windows, they need extra privileges to use them (typically an elevated prompt). So even though symlinks can be used on Windows, they typically shouldn't be.

[pfmoore]

I think you could get the build options for the Python in question from https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2

OK. With that amount of patching of the Python sources, it's no wonder you're hitting issues with virtualenv. I'm afraid I don't see that we'd accept changes to support such a heavily patched unofficial Python build.

Maybe the msys2 project would develop their own custom patched version of virtualenv, like they have done for Python?

[pe224]

Yeah, no worries. It seems an edge case. I also wonder whether this Python of use as a reliable development platform, since, as you say, it's patched a lot.
On the other hand, I had some problems trying to compile C extensions to Python with MinGW and getting them to run with a Python compiled by MSVC - so there's that.
Using Cygwin is a pain, because you need to cross-compile if you don't want to always link against cygwin DLLs.

In the end, I think the whole path problem is less about MSYS as about a Python built with MinGW64 (which is, in principle, independent of MSYS). E.g. there is mingw-builds which contains Python built with MinGW64 and will probably also not work with virtualenv. Seems like people using this config haven't discovered virtualenv yet.

For the sake fo completeness: The fix with the sitecustomize.py is not totally correct, either. If you call virtualenv.exe from cmd.exe it doesn't add the correct site-packages directory to the path, because site.py distinguishes by os.sep = '/', so in MSYS/bash case, this conditional is true and the paths are correct and he can find the sitecustomize.py. In the cmd.exe case, it's obviously flawed and he doesn't look in lib/python2.7/site-packages, but in Lib/site-packages, which doesn't exist, and he doesn't even find the sitecustomize.py. So I patched site.py as well.

Ah well :-) Thanks a lot, I got my Frankenstein working. Keep up the nice work.

[dleslie]

This is a pain; having a broken virtualenv breaks Jedi in Emacs. It's a shame that the MSYS2 team patched Python so heavily.

[iongion]

It took 3 minutes to create the env, it did not complete successful, the size of this incomplete env directory is 301 MB, for some reason it actually contains the entire include path of msys2 toolchain.
I look at this with no joy:

$ virtualenv env --no-site-packages
New python executable in W:/Workspace/Nucleus/Neuron/env/Scripts/python2.exe
Also creating executable in W:/Workspace/Nucleus/Neuron/env/Scripts/python.exe
Traceback (most recent call last):
  File "W:/Workspace/Nucleus/Neuron/env/lib/python2.7/site.py", line 703, in <module>
    main()
  File "W:/Workspace/Nucleus/Neuron/env/lib/python2.7/site.py", line 692, in main
    aliasmbcs()
  File "W:/Workspace/Nucleus/Neuron/env/lib/python2.7/site.py", line 515, in aliasmbcs
    import locale, codecs
  File "W:/Workspace/Nucleus/Neuron/env/lib/python2.7/locale.py", line 17, in <module>
    import functools
ImportError: No module named functools
ERROR: The executable W:/Workspace/Nucleus/Neuron/env/Scripts/python2.exe is not functioning
ERROR: It thinks sys.prefix is u'w:/workspace/nucleus/neuron' (should be u'w:/workspace/nucleus/neuron/env')
ERROR: virtualenv is not compatible with this system or executable
Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

[erikbsap]

so this problem is known since 2014 and still exists? Don't allow pacman -S python3-pip if you can't have virtualenvs working. Or in the installation print a warning or something to give an alternative approach.