Upgrading to distribute 0.7 (which depends on the new setuptools 0.7) ends in tears

[inducer]

This just happened:

$ pip install --upgrade distribute
Downloading/unpacking distribute from https://pypi.python.org/packages/source/d/distribute/distribute-0.7.zip#md5=93a7dea3c59cf25aa12d0026840f10ba
  Running setup.py egg_info for package distribute

Downloading/unpacking setuptools>=0.7 (from distribute)
  Running setup.py egg_info for package setuptools

Installing collected packages: distribute, setuptools
  Found existing installation: distribute 0.6.45
    Uninstalling distribute:
      Successfully uninstalled distribute
  Running setup.py install for distribute
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    Complete output from command /var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/bin/python -c "import setuptools;__file__='/var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/build/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-KbHbEQ-record/install-record.txt --single-version-externally-managed --install-headers /var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/include/site/python2.7:
    Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: No module named setuptools

To make this more precise: I get that "upgrading" distribute means uninstalling it and installing setuptools. I just wish that process wouldn't fail.

[qwcode]

the distribute project had released a new "wrapper" version of distribute that does nothing but depend on the new setuptools (the merge of setuptools and distribute).

they've since taken it down for now.
given pip's dependency on setuptools and it's install logic, pip won't be able to perform this upgrade.
distribute upgrades have only been working in python 2 anyway up until now (not python 3; #650)

decision pending on what pip's position will be on supporting upgrading to the new distribute....

[qwcode]

with distribute-0.6.X upgrades, even though pip uninstalled distribute as part of the upgrade, it ran the install subprocess with the cwd equal to the build dir, so it could import setuptools from the unpacked download.

but now, distribute-0.7 is just a shell (that depends on setuptools), that contains no importable setuptools that pip can use (and by default, setuptools gets queued to be installed after distribute)

here's a thought on adding some special logic to make this work:
(will work this out in more detail, as the work actually begins)

when an upgrade to distribute==0.7 is detected:

1. force the "setuptools" dependency to be processed before distribute itself.
2. force pip to perceive "distribute" as a conflict to setuptools (thereby forcing an uninstall of distribute first)

so the order would end up being:

• uninstall distribute-0.6.X
• install setuptools-0.7 (making use of the setuptools build dir to import setuptools)
• install distribute-0.7

cc @jaraco, @pfmoore

[jezdez]

If possible we should ship this with 1.3.x, too.

[pfmoore]

Sounds like a reasonable plan, although I can't get up any enthusiasm for it. I suspect it would still fail for Python 3, because of the unrelated 2to3 requirement/issue. That problem will go away once setuptools removes the requirement for 2to3, but I assume that won't be before 0.8. Actually, you say "when an upgrade to distribute==0.7 is detected" - should that not be ">= 0.7"? I wouldn't want to force someone upgrading to a single-source 0.8 to have to go via the 2to3 0.7...

We should keep the relevant code isolated so that it can be removed again once use of distribute has diminished to the point where it's not an issue - we don't want to have to maintain this indefinitely.

[jaraco]

That's right - a unified Python 2/3 code base won't be before 0.8, although I hope to make that release sooner than later. Don't let the number increment suggest it's anything but top priority following the merge.

And yes, I believe any distribute>=0.7 is correct. I do have plans to release 0.7.1 which will reference setuptools on pypi (removing dependency_links to get setuptools from bitbucket).

[pfmoore]

One other thought. pip 1.4 will support installation from wheels. If setuptools 0.7 is available in wheel format on PyPI, we could simply automatically force wheel installation for setuptools, and bypass the whole problem. That may be less work, and would likely involve less "hacky" code, for what is essentially an interim solution.

@qwcode, is that a possible alternative? @jaraco - it would mean hosting wheels for setuptools, is that viable?

Obviously, if we release a 1.3.x that needs to fix this issue, then this idea is a non-starter.

[qwcode]

ok, my plan is not good enough since the code is not 2&3 compatible yet. sorry, I mistakenly thought it would be. going to the trouble to fix this just for python2 is no good IMO.

4 other thoughts:

1. paul's idea of upgrading from wheels (it would require pip install --use-wheel, or installing directly from the wheel url).
2. @regebro mentioned on distutils-sig including a working 2&3 compatible source tree (if it's close to ready) that doesn't get installed.
3. "unrelease" setuptools 0.7 from pypi, and push for the 2&3 version before any more big announcements; which would mean we could use my original plan if needed
4. in addition to my plan, figure out how to also hack around the current 2to3 egg_info failure we get now in python3.

[qwcode]

for #4, we just need 2to3 to run before we try to run egg_info in python3.
shouldn't that be easy to fix? What am I missing?

[jaraco]

I'll have some time on Thursday. I'll look into merging the code patches to provide native Python 3 support and provide a report back on how close that would be.

Another option: the distribute shell package could include both the Python 2 and 2to3-converted Python 3 code plus wrapper packages/modules ('pkg_resources', 'setuptools') that selectively import the proper code. It would be hacky, but I'd be okay with that in the shell package.

As for wheel support, I'm still largely naive about wheels, so I would need to learn more about them. My suspicion, however, is that migrating all existing clients to wheels is a more challenging problem fraught with even more compatibility struggles. My goal with setuptools 0.7 is to try to remain as compatible as possible and focus primarily on migrating users from distribute back to a single project for setuptools.

[regebro]

Having two source trees is probably a good solution in this case. It makes the package big, but the install would be quicker. First running 2to3 on the distribute source and then not install it, and then install setuptools and run 2to3 on that, would be annoying and slow.

[qwcode]

to be clear, the idea of including a 2&3 importable setuptools in the distribute shell, will still leave pip failing in python3 when we get to installing the new setuptools, which is still not 2&3.

It's 2 problems:

1. needing an importable setuptools, when installing the distribute shell
2. a solution in python3 for pip and it's 2to3 egg_info failure, when installing the new setuptools

for #1, either

• jason's hack import idea he mentioned
• the hack I mentioned originally

for #2, either

• waiting on jason to get the 2&3 changes merged for a 0.8 release of setuptools
• like I mentioned, some solution that gets 2to3 run before egg_info is called. this only needs to happen for setuptools (assuming one of the 2 solutions for problem No instructions for how to install pip #1), so just one execution of 2to3, not two.

[qwcode]

@jaraco I started working on a pip-only solution for this.

one thing I can't fathom is that you can't run "python setup.py egg_info" more than once for distribute-0.7 without it getting confused somehow, and being unable to locate egg_info

qwcode@qwcode:~/downloads/distribute-0.7$ rm -rf distribute.egg-info/
qwcode@qwcode:~/downloads/distribute-0.7$ python setup.py egg_info
running egg_info
creating distribute.egg-info
writing requirements to distribute.egg-info/requires.txt
writing distribute.egg-info/PKG-INFO
writing top-level names to distribute.egg-info/top_level.txt
writing dependency_links to distribute.egg-info/dependency_links.txt
writing manifest file 'distribute.egg-info/SOURCES.txt'
reading manifest file 'distribute.egg-info/SOURCES.txt'
writing manifest file 'distribute.egg-info/SOURCES.txt'
qwcode@qwcode:~/downloads/distribute-0.7$ python setup.py egg_info
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'zip_safe'
  warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'

[regebro]

Where is the distribute source now? Tareks repo does't contain the 0.7 attempts as far as I can tell, and none of the pypa repos seem to have a version of Distribute.

The reason I ask is that I wondered if it was possible to make a Distribute 0.7 that simply did not use setuptools at all, and just distutils. I don't know, but it might be worth a try.

[pfmoore]

A thought. For the problem of needing an importable setuptools, there is a 3rd option (I think):

• Vendor in our own copy of pkg_resources
• Special-case the install command for setuptools/distribute to not go through the dance to force using setuptools, but just do python setup.py.