VirtualEnv error with Windows + Vagrant

[clone1018]

No idea where to even start with this one. Fresh clone with fresh vagrant box.

Here's the weird part, at home my Windows 8.1 x64 computer experiences this problem but my Windows 7 x64 computer at work doesn't. This also happens to @abnor

vagrant@precise64:~/www.gittip.com$ make clean env
rm -rf env *.egg *.egg-info
find . -name \*.pyc -delete
"/usr/bin/python2.7"  "./vendor/virtualenv-1.9.1.py"\
                                --unzip-setuptools \
                                --prompt="[gittip] " \
                                --never-download \
                                --extra-search-dir=./vendor/ \
                                --distribute \
                                ./env/
Traceback (most recent call last):
  File "./vendor/virtualenv-1.9.1.py", line 2577, in <module>
    main()
  File "./vendor/virtualenv-1.9.1.py", line 979, in main
    no_pip=options.no_pip)
  File "./vendor/virtualenv-1.9.1.py", line 1081, in create_environment
    site_packages=site_packages, clear=clear))
  File "./vendor/virtualenv-1.9.1.py", line 1275, in install_python
    copyfile(join(stdlib_dir, fn), join(lib_dir, fn))
  File "./vendor/virtualenv-1.9.1.py", line 437, in copyfile
    copyfileordir(src, dest)
  File "./vendor/virtualenv-1.9.1.py", line 412, in copyfileordir
    shutil.copytree(src, dest, True)
  File "/usr/lib/python2.7/shutil.py", line 206, in copytree
    raise Error, errors
shutil.Error: [('/usr/lib/python2.7/config/libpython2.7.so', './env/lib/python2.
7/config/libpython2.7.so', '[Errno 71] Protocol error')]
make: *** [env] Error 1

---

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

[patcon]

Maybe related: Scifabric/pybossa#566 (comment)

[chadwhitacre]

Blech. I looked at this thinking it might be a simple Python thing but [Errno 71] Protocol error looks more virtualization-related.

[chadwhitacre]

TeamX

[patcon]

Let's try this?
pypa/virtualenv#409

Can someone on a failing windows version give the --always-copy flag a shot?

[lukaszbanasiak]

@patcon

For me it works. Without --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env
New python executable in /vagrant/env/bin/python
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 985, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1405, in install_python
    os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error

With --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env --always-copy
New python executable in env/bin/python
Installing setuptools, pip...done.

Windows 7 64-bit, vagrant (Ubuntu 14.04 64-bit)

[chadwhitacre]

Thanks @lukaszbanasiak! So where do we need to add --always-copy? To the Makefile? Want to give us a PR for that?

[techtonik]

Just run into this. I think I'll be able to tackle this. The problem is likely related to permissions for files mounted as VirtualBox share. Git also chokes on them, because all files are executable, permissions are immutable etc.

[techtonik]

Tracing the chain so far. This is invoked from /vagrant directory.

[vagrant provision] -> [scripts/vagrant-setup.sh] -> [make env] ->
  -> [virtualenv-1.9.1.py ...]

[techtonik]

virtualenv-1.9.1 calls failing copy tree two times until fault:

shutil.copytree(src, dst, symlinks=False):

==> default: Copying /usr/lib/python2.7/lib-dynload ./env/lib/python2.7/lib-dynload
==> default: Copying /usr/lib/python2.7/config ./env/lib/python2.7/config

[techtonik]

/usr/lib/python2.7/config appears to be a directory:

drwxr-xr-x  2 root root     4096 Apr 14  2014 .
drwxr-xr-x 26 root root    24576 Apr 14  2014 ..
...
-rw-r--r--  1 root root 18965158 Feb 27  2014 libpython2.7-pic.a
lrwxrwxrwx  1 root root       23 Feb 27  2014 libpython2.7.so -> ../../libpython2.7.so.1
...

This is a symlink that can not be symlinked to /vagrant share, because it is mounted from Windows which doesn't support this.

[techtonik]

virtualenv 1.11.6 doesn't fix this. --always-copy was added in 1.10, so we need to upgrade virtualenv anyway.

[chadwhitacre]

@techtonik If you want help ping @joeyespo. He's into Vagrant + Windows.

[techtonik]

Don't need it yet, but I'll keep that in mind. Thanks. The only help needed right now is to merge some virtualenv pull requests.

[techtonik]

Filled (pypa/virtualenv#663). Going to add --always-copy for now and upgrade virtualenv to 1.11.6 to see how it goes.

[techtonik]

Didn't get far. Hit hashicorp/vagrant#4745 on the way. Shot myself in the foot with this one, actually.

[techtonik]

Next fail with virtualenv 1.11.6:

...
==> default: Cannot find a wheel for setuptools
==> default: Cannot find a wheel for pip
==> default: Installing setuptools, pip...
==> default:   Complete output from command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
==> default:   Traceback (most recent call last):
==> default:   File "<string>", line 1, in <module>
==> default: ImportError: No module named pip
==> default: ----------------------------------------
==> default: ...Installing setuptools, pip...done.
==> default: Traceback (most recent call last):
==> default:   File "./vendor/virtualenv.py", line 2338, in <module>
==> default:     main()
==> default:   File "./vendor/virtualenv.py", line 824, in main
==> default:     symlink=options.symlink)
==> default:   File "./vendor/virtualenv.py", line 992, in create_environment
==> default:     install_wheel(to_install, py_executable, search_dirs)
==> default:   File "./vendor/virtualenv.py", line 960, in install_wheel
==> default:     'PIP_NO_INDEX': '1'
==> default:   File "./vendor/virtualenv.py", line 902, in call_subprocess
==> default:     % (cmd_desc, proc.returncode))
==> default: OSError: Command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1
==> default: make: *** [env] Error 1

Looks like pip upgrade is needed.

[techtonik]

Downloading pip and setuptools wheels helped. Going to commit later.

[techtonik]

Well, another problem:

...
  File "distribute_setup.py", line 129, in _do_download
    _build_egg(egg, tarball, to_dir)
  File "distribute_setup.py", line 120, in _build_egg
    raise IOError('Could not build the egg.')
IOError: Could not build the egg.

[joeyespo]

RE: Vagrant + Windows

Honestly, even after fixing the quirks, working within a Vagrant environment hasn't been a pleasant dev experience. It's terribly slow. And getting set up is still quite tedious (which is the whole point of Vagrant). This has kind of pushed me away from participating. I don't feel productive because of how much time it takes to get from "idea I want to test" or "issue I want to resolve" to "localhost".

Instead, I'd much rather work within a "native" Python and JavaScript dev environment since these technologies are already cross-platform. Striving for a single, cross-platform run step (and ideally, a single, cross-platform install step too) is a much better goal than "let's fix Vagrant."

I've applied this approach to other websites (both monolithic and microservices), and it's made both onboarding and day-to-day development much smoother. Contributors can continue to use the tools and the environments that make them maximally productive. And there's no time wasted with installation hand-holding. Requiring each machine to be exactly the same in order to make high-level changes shuts out a lot of people. (As for the dev/prod-parity as a bug-catching mechanism, which usually comes up, auto-running tests in a prod-like CI tool before merging PRs has been equivalently effective.) I'd be happy to talk more about the approaches I've used to make webapps more cross-platform if there's any interest here, but unfortunately, I don't think I'll be much help with Vagrant issues anymore.

[chadwhitacre]

Thanks for checking in, @joeyespo! So are you saying Vagrant in general is a buzzkill for you lately, or Vagrant as we've implemented it? I think I'm hearing you say the former.

[joeyespo]

@whit537 Sure! Glad things are going strong. I also really like the rename, it's catchy :-)

Yeah, the former.

[chadwhitacre]

@joeyespo So what's your secret formula? Let me guess: use SQLite? ;-)

[techtonik]

IOError: Could not build the egg. is caused by old distribute conflicting with new setuptools.
https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older

[techtonik]

This one is blocked by #2886 through #2867 ATM..

[joeyespo]

@whit537 Haha, yeah, that's one option. A decent one too, since it's built-in to Python and is supported by popular ORMs.

The general idea is use cross-platform auto-installable components for dev, and production-optimized components for prod, staging, and tests running on CI.

Since Python/JS packages are already auto-installable and cross-platform, it's a natural fit to use them in place of the prod-ready counterparts. Of course, not every prod-ready component has a pure-Python implementation. In those cases, you have to substitute higher up. ORMs are a good option because the ability to plug in different backends is already built for you.

As for postgres.py, I'm not sure what the best approach would be. It highly couples your app to Postgres instead of to an object model. But I'm willing to talk through some ideas. Let's move the discussion to a new issue.

[techtonik]

Bootstrap fails with DB error:

==> basebox: env/bin/honcho -e defaults.env,local.env run env/bin/fake_data fake_data
==> basebox: Traceback (most recent call last):
==> basebox:   File "env/bin/fake_data", line 9, in <module>
==> basebox:     load_entry_point('gratipay==1736.post27', 'console_scripts', 'fake_data')()
...
==> basebox: psycopg2.IntegrityError: new row for relation "transfers" violates check constraint "positive"
==> basebox: DETAIL:  Failing row contains (578395790, 2014-08-20 12:00:37+00, Dominicbdp, Katharina92q, 0.00, tip).
==> basebox: make:
==> basebox: *** [data] Error 1

Moved to #2950

[techtonik]

Please test new vagrant up and create tickets for any issues you may run into. I think now it is good on Windows.