Make typing and typed-ast external dependencies #2452

gvanrossum · 2016-11-14T22:01:03Z

Reviving #2340.

This will automatically install the typing and typed-ast packages when
appropriate.

The setup.py file now depends on setuptools. We should build wheels
so that users installing from PyPI won't need setuptools. In order to
build wheels the distribution builder/uploader needs to install the
wheel package. To manage those dependencies, I've added
build-requirements.txt.

In summary:

python3 -m pip install -r build-requirements.txt
python3 setup.py bdist_wheel
upload the .whl file that appears in the dist/ subdirectory to PyPI

Remaining issues:

The installed scripts (mypy, stubgen) have the Python version used to create the wheel file hard-coded, so I'll have to do some more digging.
I'm getting warnings about the version being invalid.

gvanrossum · 2016-11-15T18:06:50Z

Hm. I cannot repro the failure at home, but I suspect that what's going on is that the version seen by the test harness is now different from the version used by the installed mypy, in particular I think the test harness sees '0.4.6-dev-$HASH' while the installed mypy uses '0.4.6-dev'. But why would this fail for Python 3.3, 3.4 and 3.5.1, but pass for Python 3.6-dev? That's still a mystery.

gvanrossum · 2016-11-15T18:15:06Z

One possible difference: in the (failing) 3.5.1 build, the install logs say

Collecting lxml (from -r test-requirements.txt (line 2))
  Downloading lxml-3.6.4-cp35-cp35m-manylinux1_x86_64.whl (4.2MB)

while in the successful 3.6-dev build they say

Collecting lxml (from -r test-requirements.txt (line 2))
  Downloading lxml-3.6.4.tar.gz (3.7MB)

and later

  Running setup.py bdist_wheel for lxml ... done

gvanrossum · 2016-12-09T20:15:53Z

I've rebased and added comments clarifying why the dependencies must be specified both in setup.cfg and in setup.py, and when exactly setuptools is needed.

Maybe we should also add a comment to the top of setup.py, or perhaps to README.md, explaining how to build a wheel file for distribution via PyPI (since it's hard to find docs on how to do this with Google search).

gvanrossum · 2016-12-09T20:30:39Z

So there's still that weird test failure for all Python versions except 3.6-dev that I cannot reproduce locally. See #2452 (comment) above.

ambv · 2016-12-12T19:56:16Z

setup.py

+# This requires setuptools when building; setuptools is not needed
+# when installing from a wheel file (though it is still neeeded for
+# alternative forms of installing, as suggested by README.md).
+from setuptools import setup


This causes $VERSION to be set as an environment variable, hence cobertura tests fail. I'm looking into how to fix this cleanly. Also, I don't quite understand why 3.6 tests passed.

Really? By what mechanism does $VERSION get set?

I am talking nonsense. What happens is as follows:

the XML report template in test-data/unit/cmdline.test has $VERSION and $TIMESTAMP abstracted away because they will obviously be different too often

the code responsible for it is in mypy/test/testcmdline.py, line 101

however, with setuptools the test runner's mypy.__version__ is "0.4.7-dev-2c9250eb182708402c66f8197f62ebe0190a43d5-dirty" so it doesn't replace the "0.4.7-dev" string in the XML

I'm investigating now why this is the case.

The test harness does python setup.py install which is different on setuptools from python setup.py develop (which is the equivalent of pip install -e .).

There's two ways forward to fix this:

Since local developers are likely going to keep using pip install -e . or equivalent, we can simply change the command that says python setup.py install to say python setup.py develop. I don't like this because this makes the cmdline test less isolated (we never perform a complete installation).

Alternatively, let's just add line 102 to mypy/test/testcmdline.py which performs another replace to $VERSION, but this time using mypy.version.base_version and not mypy.version.__version__. This makes it work for both cases and won't break when we implement Version doesn't include git commit hash when installed via "pip3 install ." #2547.

OK, I think about as far as I got the previous time around. The big mystery is why 3.6-dev behaves differently. I noticed that there was something different in the way lxml is installed. See previous comments (search for lxml).

I'm for (2). I'm not sure what pip install -e . or python setup.py develop do (the help text is kind of minimal) so (1) is particularly unattractive for me.

Why setuptools would behave differently on 3.6, no idea. lxml 3.6.4 is a red herring, the difference is because there is a lxml 3.6.4 wheel pre-built for 3.5 on PyPI but none for 3.6. This went away BTW since there's just .tar.gz for lxml 3.7.0.

I can confirm both of my suggested fixes solve the failure.

ambv · 2016-12-12T19:59:09Z

setup.py

+install_requires = []
+if sys.platform != 'win32':
+    install_requires.append('typed-ast >= 0.6.1')
+if sys.version_info < (3, 5):


if/else statements like these produce incorrect wheels, e.g. dependent on the Python version with which setup.py was called. Instead, you should say:
extras_require={':python_version<"3.5"': ['typing >= 3.5.2']},
inside the setup() call.

How certain are you of that? IIRC I tested this and I found that the dependencies in the wheel are determined by what's in setup.cfg, not by the code here. But I am certainly mostly cargo-cult-coding here, since I haven't found decent docs on how to do these things. (Just StackOverflow questions, and various bits of doc that recommend things that don't work when I try them.)

I got a report to that effect on my configparser backport: https://bitbucket.org/ambv/configparser/pull-requests/1/use-specifiers-to-indicate-dependence-on/diff

You might be right that setup.cfg is enough, I'll play with this to confirm/deny.

ambv · 2016-12-12T19:59:42Z

setup.py

@@ -103,10 +115,11 @@ def run(self):
      license='MIT License',
      platforms=['POSIX'],
      package_dir=package_dir,
-      py_modules=['typing'] if sys.version_info < (3, 5, 0) else [],


I don't understand why this is here.

You mean you don't understand why it was there right? mypy itself depends on the typing module. Previously mypy just installed a copy of typing.py (lib-typing/3.2/typing.py), but only for Python 3.4 and earlier, where it's not in the stdlib -- for Python 3.5 it depends on the stdlib (and installing another copy wouldn't have any effect anyway).

OK, that explains it. I'm just used to py_modules being defined on the packages that implement a feature, not for dependencies.

gvanrossum

Thanks for helping! We are going to need this for the next mypy release (when typed_ast is required so we can make --fast-parser the default) and nobody here knows much about setup.py.

gvanrossum · 2016-12-12T21:20:58Z

setup.py

+# This requires setuptools when building; setuptools is not needed
+# when installing from a wheel file (though it is still neeeded for
+# alternative forms of installing, as suggested by README.md).
+from setuptools import setup


Really? By what mechanism does $VERSION get set?

gvanrossum · 2016-12-12T21:22:52Z

setup.py

+install_requires = []
+if sys.platform != 'win32':
+    install_requires.append('typed-ast >= 0.6.1')
+if sys.version_info < (3, 5):


How certain are you of that? IIRC I tested this and I found that the dependencies in the wheel are determined by what's in setup.cfg, not by the code here. But I am certainly mostly cargo-cult-coding here, since I haven't found decent docs on how to do these things. (Just StackOverflow questions, and various bits of doc that recommend things that don't work when I try them.)

gvanrossum · 2016-12-12T21:24:55Z

setup.py

@@ -103,10 +115,11 @@ def run(self):
      license='MIT License',
      platforms=['POSIX'],
      package_dir=package_dir,
-      py_modules=['typing'] if sys.version_info < (3, 5, 0) else [],


You mean you don't understand why it was there right? mypy itself depends on the typing module. Previously mypy just installed a copy of typing.py (lib-typing/3.2/typing.py), but only for Python 3.4 and earlier, where it's not in the stdlib -- for Python 3.5 it depends on the stdlib (and installing another copy wouldn't have any effect anyway).

gvanrossum · 2016-12-12T21:35:33Z

FWIW I would love not to have to use setuptools. Mypy is an app, not a library, and we only really care about using pip to install it. These are what we want to support, and all should automatically install the dependencies too:

pip install mypy-lang # official version from PyPI
pip install . # from current repo
pip install git+git://github.com/python/mypy.git # master from GitHub

gvanrossum · 2016-12-12T23:16:46Z

FWIW I fear that using base_version anywhere outside setup.py is going to lead to disappointments, because the way mypy/version.py is overwritten by setup.py means that it won't exist when running a version installed from the installer.

ambv · 2016-12-12T23:22:38Z

setup.py doesn't even include mypy/test/* and test-data/* so making mypy/test/testcmdline.py depending on base_version shouldn't break. If you have strong concerns, you can make this into a no-op by replacing the base_version import with the try:except ImportError: dance. But as I said, I don't think we need to worry about this in tests.

gvanrossum · 2016-12-12T23:25:21Z

Travis CI confirms that, so I will stop worrying.

Can you summarize other stuff you think I should change?

ambv · 2016-12-12T23:40:22Z

I can confirm that setup.cfg properly produces py3-any wheels that include proper environment-specific requirements in metadata.json and METADATA files. So nothing needs to change here.

One nit, since we are no longer installing lib-typing and wheels don't include them, maybe we should just delete lib-typing from the repo? Esp. that the sdist is including lib-typing but not using it anymore due to your changes to setup.py. There's some hackery about it in the test runner and mypy/test/testpythoneval.py so this might better fit a separate PR.

LGTM.

gvanrossum · 2016-12-12T23:55:00Z

Thanks for your help! I have to think about the lib-typing issue some more -- IIRC it's still referenced by some tests. Also before making this live I need to get someone to test the wheel it produces on Windows.

…l on platform/version (#2573) Also make typed_ast and typing conditional on platform/version. I found that typing was missing when attempting to run pytest in a mostly vanilla 3.3 install. The platform/version conditionals are borrowed from PR #2452.

ethanhs · 2016-12-23T01:47:11Z

Okay so from linux running python3 setup.py bdsit_wheel I then copied the wheel to my Windows partition and pip installed it. MyPy seems to work fine. It ran through everything in test-data/samples fine, and with (code from two issues raised as examples of what should fail):

a = b'%b' % b'123'
assert a == b'123'

from typing import Iterator

class Foo(object):
    def __iter__(self) -> Iterator[int]:
        yield 1
        yield 2

for x in Foo():
    print(x)

It correctly outputted :

test_484.py:1: error: Unsupported format character 'b'
test_484.py:11: error: Iterable expected

So everything looks good to me!

gvanrossum · 2016-12-23T18:39:39Z

@JukkaL Do you have additional concerns about this?

This will automatically install the typing and typed-ast packages when appropriate. The setup.py file now depends on setuptools. We should build wheels so that users installing from PyPI won't need setuptools. In order to build wheels the distribution builder/uploader needs to install the wheel package. To manage those dependencies, I've added build-requirements.txt. In summary: - python3 -m pip install -r build-requirements.txt - python3 setup.py bdist_wheel - upload the .whl file that appears in the dist/ subdirectory to PyPI

…cific typing package

gvanrossum · 2017-01-06T20:32:31Z

This now needs a few more things before it can be merged:

A new typed_ast that fixes the compile error with Python 3.3 (Linux only) and pushed to PyPI
Updates to setup.py and setup.cfg to incorporate the new typed_ast version
An end-to-end install test on Windows

ethanhs · 2017-01-06T20:34:27Z

@gvanrossum could you clarify what you mean by "end to end" happy to install mypy from source and test if that is what you mean.

gvanrossum · 2017-01-06T20:36:59Z

What you said -- David is also going to try this.

ethanhs · 2017-01-06T21:00:02Z

Okay, built from source, a few tests failed (testcmdline, mypy.test.testcheck, mypy.test.testextensions). For testcmdline, some of the paths are hard-coded, and so with Window's \ instead of /, it failed, I went through and manually verified that the results were the same except for the directory separators. The other two said they did not have tests. Here is the full log. I will take a look at why the other two are failing later today.

EDIT: looks like David was successful, must be doing something wrong on my end...

ambv · 2017-01-06T21:02:27Z

@ddfisher Could you also build typed_ast 0.6.3 wheels for Linux on Python 3.5 and 3.6? This will speed up Travis CI.

ddfisher · 2017-01-06T21:03:50Z

@ethanhs: Thanks for looking into it! Those tests have probably been failing for a while, right? Are they necessarily related to this PR?

@ambv: I'll look into getting a Travis build set up for that.

ethanhs · 2017-01-06T21:15:04Z

@ddfisher As I recall when I first ran the tests by building on Linux and installing on Windows, all of the tests passed when I ran tests on Linux. That was 15 days ago, so perhaps they are failing now...

ethanhs · 2017-01-06T21:31:13Z

Just ran the test suite on Linux and it passed everything, however it seems the two failing tests were not run. So LGTM. At some point we'll want to fix the hardcoded test or just turn off all three.

ddfisher · 2017-01-06T22:57:13Z

@ambv: I spent some time trying to get a manylinux build for typed_ast running on Travis but have only met with partial success. There are some higher pri things for me to work on (like actually making typed_ast mypy's default parser), so I'm not going to spend more time on it right now. I'll probably give it another look in a week or two. In the meantime, PR's welcome! (My partial progress is here: ddfisher/typed_ast@99590f0)

gvanrossum mentioned this pull request Nov 14, 2016

[WIP] Make typing and typed-ast external dependencies #2340

Closed

gvanrossum force-pushed the reqs branch 3 times, most recently from fae1c5c to 4d66f81 Compare November 15, 2016 17:21

gvanrossum force-pushed the reqs branch 3 times, most recently from b4f83eb to 8daa9ee Compare November 16, 2016 17:36

gvanrossum force-pushed the reqs branch from 8daa9ee to 2c9250e Compare December 9, 2016 20:13

gvanrossum mentioned this pull request Dec 9, 2016

Version doesn't include git commit hash when installed via "pip3 install ." #2547

Closed

ambv reviewed Dec 12, 2016

View reviewed changes

gvanrossum commented Dec 12, 2016

View reviewed changes

gvanrossum mentioned this pull request Dec 13, 2016

Add typing to test-requirements #2573

Merged

ambv mentioned this pull request Dec 22, 2016

Consider switching to setuptools #482

Closed

Guido van Rossum added 4 commits January 6, 2017 12:17

Make Python 3.3/3.4 tests pass, by skipping a test depending on a spe…

c219961

…cific typing package

Update setup.cfg with cargo-culted stuff

263f005

Our wheels are not universal

7d91652

Guido van Rossum and others added 4 commits January 6, 2017 12:17

Add comments clarifying some things added here

08c86df

Attempt to fix test failure by also replacing base_version with $VERSION

8e8298d

Update setup.cfg

2ab1e6e

Update setup.py to match setup.cfg

b8468e7

gvanrossum force-pushed the reqs branch from 16dcd1f to b8468e7 Compare January 6, 2017 20:18

We need typed_ast 0.6.3

09d39c7

ddfisher merged commit a207754 into master Jan 6, 2017

ddfisher deleted the reqs branch January 6, 2017 21:00

ddfisher mentioned this pull request Jan 13, 2017

switch to setuptools and provide wheels. python/typed_ast#3

Closed

JukkaL mentioned this pull request Jan 16, 2017

Fail if an old typed_ast version is installed #2384

Closed

Make typing and typed-ast external dependencies #2452

Make typing and typed-ast external dependencies #2452

Conversation

gvanrossum commented Nov 14, 2016

gvanrossum commented Nov 15, 2016 • edited Loading

gvanrossum commented Nov 15, 2016

gvanrossum commented Dec 9, 2016

gvanrossum commented Dec 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ambv Dec 12, 2016 • edited by gvanrossum Loading

Choose a reason for hiding this comment

ambv Dec 12, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented Dec 12, 2016

gvanrossum commented Dec 12, 2016

ambv commented Dec 12, 2016

gvanrossum commented Dec 12, 2016

ambv commented Dec 12, 2016

gvanrossum commented Dec 12, 2016 via email

ethanhs commented Dec 23, 2016 • edited Loading

gvanrossum commented Dec 23, 2016

gvanrossum commented Jan 6, 2017 • edited Loading

ethanhs commented Jan 6, 2017

gvanrossum commented Jan 6, 2017 via email

ethanhs commented Jan 6, 2017 • edited Loading

ambv commented Jan 6, 2017

ddfisher commented Jan 6, 2017

ethanhs commented Jan 6, 2017 • edited Loading

ethanhs commented Jan 6, 2017

ddfisher commented Jan 6, 2017

gvanrossum commented Nov 15, 2016 •

edited

Loading

ambv Dec 12, 2016 •

edited by gvanrossum

Loading

ambv Dec 12, 2016 •

edited

Loading

ethanhs commented Dec 23, 2016 •

edited

Loading

gvanrossum commented Jan 6, 2017 •

edited

Loading

ethanhs commented Jan 6, 2017 •

edited

Loading

ethanhs commented Jan 6, 2017 •

edited

Loading