pip-compile on 1.10.2+ fails to handle mypy; >= '3.2' on python 2.7

[tuukkamustonen]

I believe this is related to how environment markers are handled.

With requirements.in of:

mypy >= 0.560; python_version >= '3.2'

On python 2.7, run:

pip-compile --no-index --rebuild -v requirements.in

Produces:

Using indexes:
  ---snip---

                          ROUND 1                           
Current constraints:
  mypy>=0.560

Finding the best candidates:
  found candidate mypy==0.560 (constraint was >=0.560)

Finding secondary dependencies:
  mypy==0.560 not in cache, need to check index
Traceback (most recent call last):
  File "/home/musttu/.virtualenvs/wires/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 187, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/resolver.py", line 199, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/resolver.py", line 285, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/repositories/local.py", line 64, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 157, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/musttu/.virtualenvs/wires/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmpreRyRhbuild/mypy/

This works on 1.10.0 and 1.10.1 but not on 1.10.2+ or 1.9.x.

On python 3.5 and 1.11.0, it works, and output even retains the environment marker:

...
mypy==0.560 ; python_version >= "3.2"
...

Environment Versions

1. OS Type: linux
2. Python version: 2.7.12
3. pip version: 9.0.1
4. pip-tools version: 1.11.0

[vphilippon]

Hello @tuukkamustonen, sorry for the delay.

You are right, this is due to how we handle the markers in the requirement file. pip-compile currently has a very partial support for environment markers, mostly for happy cases.

This is a known issue; pip-compile currently doesn't interpret environment markers in a requirement file itself. It simply brings them over to the output file. As pip-compile ends up processing mypy's setup.py on python 2.7, you get this result.

The reason it "worked" under v1.10.0 and v1.10.1 was actually due to a bug where the python 3 wheels would be used even on python 2 to get the dependencies (effectively meaning you were getting the python 3 variant of the dependencies, which may or may not be different). As a result, the .tar.gz was never used, the setup.py not executed, and you weren't getting any issue (beside getting the dependencies list of a python 3 wheel while on python 2, of course).

This is something I would like to see fixed in the future of course, but I'm bit short on time.
Contributions are always welcomed 😄

[onyb]

@vphilippon I had a similar issue while trying to install jsonschema==2.6.0 inside a Python 3.6 virtualenv. If you look at the install_requires section of jsonschema's setup.py file here, you'll find this:

install_requires=[
    "attrs>=17.3.0",
    "pyrsistent>=0.14.0",
    "six>=1.11.0",
    "functools32;python_version<'3'",
],

pip-compile is unable to ignore the functools32 dependency under Python 3.6, and I get the following error while installing from the requirements.txt: "This backport is for Python 2.7 only."

Could you please point me to the specific sections of the codebase where I can take a look for investigating the bug, so that I can submit a PR?

[vphilippon]

Hi @onyb, thanks for your offer!

I can point you to https://github.com/jazzband/pip-tools/blob/master/piptools/scripts/compile.py#L187 and https://github.com/jazzband/pip-tools/blob/master/piptools/resolver.py#L74

Although, the solution might be be to do something similar to https://github.com/jazzband/pip-tools/blob/master/piptools/sync.py#L114 (use .match_markers) to filter packages based on the system.

But we have to be carefull with to not break the generate_hash logic that generates hashes for all platforms.
That being said, filtering dependencies based on the current platform could also cause some issue with the work done in #460.

If we mix the 2, what I forsee is that we will get in a state where we compute dependencies only for the dependencies of the current system/environment, while introducing potential "unresolved" packages in the final file, essentially breaking the reproductibility promise.

If we want to support the workflow where you have a single requirements.in file with markers, with which you perform a pip-compile on each target environment (python version + OS), then we would have to revert #460, as it appears simply incompatible to me.

@barrywhart @suutari-ai @davidovich Any input here?

[barrywhart]

I have had similar issue with functools, where it was being included as an indirect dependency of an application that needed to support both Python 2.7 and 3.x. In that case at least, there is a straightforward workaround or solution. Include the following line in requirements.in:

functools32 ; python_version=='2.7'

The environment marker is passed through to requirements.txt even though the library requiring it doesn't include the marker, e.g.

functools32==3.2.3.post2 ; python_version == "2.7"

But your case is interesting, because jsonschema is actually specifying the environment marker. Are you saying it generates a requirements.txt file with functools32 but without the environment marker? If so, that sounds like a bug in #460, and perhaps a different issue than the original poster on this issue.

pip-compile is a very useful tool, but it also seems subject to occasional breakage because it's trying to do something that isn't really well supported by the underlying tools. That is, there's no universal, cross-Python version/environment way to get dependency information for a package.

[barrywhart]

FWIW, I can't reproduce the jsonschema issue. I created a Python 3.6.3 virtualenv and ran pip-compile using the following requirements.in:

jsonschema==2.6.0

And it generated the following requirements.txt:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
jsonschema==2.6.0

[vphilippon]

@onyb i did the same check as Barry, and couldn't reproduce either. What's the setuptools version in the virtualenv? Older version did not interpret markers in the install_requires. Otherwise, check if functools32 might be coming from another package which doesn't have the environnement marker.

--

[...] it's trying to do something that isn't really well supported by the underlying tools. That is, there's no universal, cross-Python version/environment way to get dependency information for a package.

@barrywhart You summarized the whole pain of trying to deal with cross-environment dependency resolution. Thank you 👍

Regarding your functools32 example, I tried to pip-compile under python3, and it fails due to being python2 only. That is because the environment marker doesn't prevent the attempt of dependency resolution of functools32.
But, if I perform the pip-compile on python2, then it will succeed, and I would have a requirements.txt
with the environment marker on functools32.
In other words, the pip-compile has to be done in an environment where all packages are installable to succeed.

My honest feeling is that pip-tools is never going to be able to generate a true cross-environment requirements.txt. The main reason is the non-deterministic aspect of setup.py, as described here: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies

I think that a clear statement regarding this should be made: That the vision of pip-tools is to generate a requirements.txt that is complete and reproducible for the environment on which it was generated (with the possibility of an exception of the generate-hashes, for the happy case where the dependencies are the same across environments). This means that a requirements.in should be pip-compile'd on each target environment individually. With that said, we can improve pip-tools by having it parse the environment markers in the requirements.in, as pip, would and include/exclude the top-level dependencies from the dependency resolution. That would effectively exclude it from the final requirements.txt too.

Sadly, this wouldn't cover the case where a lib forgets to put environment markers on a package, as given by @barrywhart . The "real" answer is to have the offending package fix its dependencies, as it would with any version conflicts. The only other way around that would be to keep the environment markers from requirements.in in the requirements.txt as we do now, and in this particular instance the python2 version of the requirements.txt would work on python3, if there's no other variation. Maybe this could be added as an option instead to avoid unneeded markers for users that follow the suggested flow.

To be a 100% honest, I would prefer this feature (copying environment markers) to be removed. I hate to take away any contribution to the Open-Source (heck, we need more!), While it appeared to be a first step toward a better solution at first, I fear that keeping this feature would give us a lot of headache and a lot more trouble with this change now. I had to deal with a similar behavior when offering help and support with Pipenv, and I'm absolutely not willing to offer the same help and support to users here; It was really rough and confusing for everyone. I also doubt pip-tools has enough other knowledgeable contributors to take care of that user support.
Here is a are some examples of issues users would likely face and request support for (or worse, not be aware of at all):

• Users would use the option to keep the markers, and then be surprised to see th package disappear from the requirements.txt when pip-compileing on another environment.
• Users would use the option to keep the markers and put a marker on a package with dependencies of its own that are possibly not required on the current environment, be lead to believe that they have a "happy case" where the requirements.txt is cross-environment, and then proceed to install on another environment, resulting in a non-deterministic installation with packages not listed in the requirements.txt, breaking one of pip-tools base promises.
• Users would use the option to keep the markers and put a marker on a package with dependencies of its own that are possibly required only on the current environment, be lead to believe that they have a "happy case" where the requirements.txt is cross-environment, and then proceed to install on another environment, resulting in installing packages that are meant for another environment.

@barrywhart If you have a maintainable idea to keep that feature to un-break some cases as you mentioned, I would be happy to hear it.

Of course, such a change would be done as part of a Major release.

(That was quite a wall of text. There was a lot to point out. I really wish this was easier to deal with.)

[barrywhart]

Perhaps we could keep the environment marker copying, but make it optional (i.e. the default is to not propagate them).

I see your point about people getting confused using it, but TBH, I wonder how many people who use pip-compile understand that it's intended to target only the environment where it was built. I suspect most people just try it out and only look at the details when something surprises them.

@vphilippon, can you clarify your question here:

@barrywhart If you have a maintainable idea to keep that feature to un-break some cases as you mentioned, I would be happy to hear it.

Are you referring to something I said on this ticket, or is it a general question?

[vphilippon]

@barrywhart That was a general question. And of course, I intend to better document and put upfront the idea that pip-tools aims to take care of a single environment.

I'll let the idea of keeping the optional marker-copy behavior running in my head for a while, see how it comes out.

[barrywhart]

Idea: Just as pip-compile includes the original command in the requirements.txt file, e.g.:

# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --no-index --output-file requirements.txt requirements.in
#
cachetools==2.0.1         # via google-auth
certifi==2018.1.18        # via requests
...

It could also include details on the environment. pip-compile should fail if the user runs it on a different environment (i.e. if the requirements.txt file already exists and the runtime environment does not match the environment listed in requirements.txt).

[barrywhart]

Idea: Provide a tool similar in spirit to tox which generates requirements.txt files for a specified set of environments (i.e. Python interpreter versions, CPython or PyPy). Having an automated process for creating a file per environment makes cross-environment files seem less important.

[vphilippon]

Todo: Document the expectation that pip-compile should be executed for each target environment.

@barrywhart For your first idea, that wouldn't play too well with the --no-header option. Also, I'm not too keen on adding additional things like that: I'd like the requirements.in/requirements.txt files to stay "pure pip requirements file". I like the idea of trying to warn/prevent a user trying to run a file on the wrong environment, but I think this kind of "advanced" stuff would be better suited with pipenv.

For your second idea, that sounds like a good tool idea. I wonder how well it could be done simply with tox without any additional work here (except maybe documented examples), or how well it would stand as a standalone tool made to deal with other cross-environment tasks. I have a hunch that we'd be better off with a distinct tool, but I might be biased as a kind-of-busy-maintainer.

[tuukkamustonen]

Oopsie, accidentally closed it :)

Though, it really should be closed now that #647 is in. Will give 2.x a spin tomorrow!