Support for Jython 2.7rc2+

[jamadden]

Jython 2.7 has to use its bundled version of pip as detailed in this bug and expounded upon in detail in this bug. Attempting to install or use newer versions simply fails.

This PR awkwardly makes that happen. Most notably, there is no way to install only setuptools but not pip...because installing setuptools depends on putting the newer bundled copy of pip on the PYTHONPATH, which fails.

Without a working virtualenv on Jython, tox can't work either, and without a working tox many projects won't bother to test on Jython. I didn't include changes to tox.ini for that reason: it needs a newer release based on a newer release of virtualenv before it can work.

If the upstream Pip is changed, then this 'if' block can be dropped in favor of adding JYTHONPATH alongside PYTHONPATH in install_wheel (because Jython only reads from that environment variable).

[ionelmc]

Wouldn't it make more sense to fix the issue in pip/html5lib (eg: integrate the changes from https://github.com/jythontools/pip) than have this workaround?

[jamadden]

Indeed, and I hope that's ultimately the plan. I know this is yet more implementation-specific code and it's not elegant. But Jython 2.7rc2+ has a modified custom version of the 'ensurepip' module already in place, and I suspect that's probably not going to change for the 2.7.0 release...if the changes can get integrated back into Pip, and Pip released, it might be able to make it on the roadmap for 2.7.1? @jimbaker would probably be the one to know about and comment on those things.

I'm just trying to help, as an interested user, get as much of the ecosystem ready as I can; I know the core Jython team is really busy trying to get the final release out ASAP.

[tseaver]

@ionelmc The pip maintainers turned down patch their bundled version of requests / html5lib to solve the issue in html5lib, which is stalled.

[jimbaker]

Jason, thanks for bringing this up. We are not going to change our bundled pip/setuptools wheels for 2.7.0, but we can workaround on our side for a future release: I have opened up a bug in Jython to support isolated surrogates in unicode objects, just so we can avoid this problem in this and other libraries http://bugs.jython.org/issue2340. I'd like to see this completed for Jython 2.7.1. The key of this approach will be to use UCS4 encoding for such unicode objects, plus forbidding interoperation with Java:

    @Override
    public Object __tojava__(Class<?> c) {
        if (c.isAssignableFrom(String.class)) {
            throw Py.TypeError("Cannot convert unicode with isolated surrogates to Java");
        }
        return super.__tojava__(c);
    }

I just started playing with this idea, using a hacky approach, and it seems to work: in Python code only, it's fine, one can freely use them like other unicode objects, and when it hits Java calls, it fails cleanly with a TypeError. A clean approach per https://www.python.org/dev/peps/pep-0393/ is probably the way to go.

[jamadden]

Thank you Jim. I looked at your bug report and it looks like an interesting approach.

As for this issue, and ecosystem support for 2.7.0: I seem to recall that you mentioned moving Jython to a time-based release schedule in your PyCon talk, but I don't remember any more details. Is that right, and if so, do you know if there is an estimated date for 2.7.1? If virtualenv expects to release before that time, it would still be very helpful to be able to make it, and tools that use it, like tox, work on 2.7.0. (Many of the projects I contribute to have the policy of not officially supporting Jython until it can be tested by tox.)

[jamadden]

Well, I say that, but it now looks like there's been some motion on the html5lib/pip issue @tseaver pointed to that's the reason jython has to bundle it's own pip. There's a new PR open at html5lib/html5lib-python#185.

[gsnedders]

Turns out prodding me is quit effective when I've giving up on solving the blocking issue. :)

html5lib master should now run on Jython (though there's no CI for it, so it may break…), and hopefully will get a release out by the end of the week (probably tomorrow).

[jimbaker]

I haven't followed this development closely enough, but is this workaround still necessary given the updates to html5lib-python, now at version 0.99999? (My understanding is no, and this PR can safely be closed out without merging.)

[tseaver]

Because pip vendors-in requests, which vendors in html5lib, we actually need releases of both those projects which either a) drop the vendoring (hell will freeze over first, AFAICT), or b )re-vendor the new upstream html5lib.

[Ivoz]

AFAIK this should be fixed implicitly with pip 7+ thanks to its vendored html5lib fixing jython support itself