-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pipenv lock
nondeterminism with environment markers
#4967
Comments
@jfly Thanks for your report! I have just did some triage of it and I agree with what you are saying that this affects latest published version
I actually have a branch I have been working on that upgrades pipenv to pip 22.x but there are still 2 failing unit tests (not convinced they are real world failures based on my debugging) and the other issue is some of the changes are in the upstream vendor code (not many). That being said, I tried your example in both I generated the lock file fresh twice in each version and then diffed the files:
Basically what that shows is the updated resolver in pip 22.x is yielding a consistent result between my two test runs where clearly the prior version did not. getting to pip 22.x would also fix some other resolver related bugs including the infinity lock bug as far as I can tell. Won't be easy to get across the line in some respects, it means dropping support for python 3.6, and getting update to requirementslib, pipshims, and pip itself currently. If you have a chance, please take a look at that PR and give the branch a try if you have time to help validate if it improves the issue from your perspective. Thanks! |
We will be dropping support for python3.6 soon, and there is a path forward to pip 22.x vendoring, so this will be fixed hopefully end of April or early May. |
Please check with |
Sorry for the radio silence here. I just tried this with |
For the record, this bug seems to be occurring again. In fact, I think it never actually was fixed =( Since I don't have the power to re-open issues, I've filed #5239 |
Issue description
I'm seeing behavior where repeated
pipenv lock
s produces differentPipfile.lock
s.One way to reproduce this problem is with a
Pipfile
that depends on bothgevent
andsqlalchemy
: example Pipfile.gevent and sqlalchemy both depend on greenlet, but they use different environment markers:
gevent 21.12.0
depends ongreenlet >= 1.1.0, < 2.0; platform_python_implementation=="CPython"
: https://github.com/gevent/gevent/blob/21.12.0/setup.py#L196-L205sqlalchemy 1.4.31
depends ongreenlet != 0.4.17;python_version>='3' and (platform_machine=='aarch64' or (platform_machine=='ppc64le' or (platform_machine=='x86_64' or (platform_machine=='amd64' or (platform_machine=='AMD64' or (platform_machine=='win32' or platform_machine=='WIN32'))))))
: https://github.com/sqlalchemy/sqlalchemy/blob/rel_1_4_31/setup.cfg#L44Actual result
I haven't really dug into the pipenv source code to understand the source of the nondeterminism, but if you look at the
pipenv lock --verbose
, you can see that one example starts withReporter.adding_requirement(SpecifierRequirement('gevent'), None)
, and the other starts withReporter.adding_requirement(SpecifierRequirement('sqlalchemy'), None)
. I'm not sure what the source of that difference in behavior is (perhaps things are getting put in a Pythonset
, and then getting iterated over, and python doesn't make any promises about the order of iteration there?)Lock produces greenlet with gevent's markers
Lock produces greenlet with sqlalchemy's markers
Expected result
I'm pretty sure that in this case, sqlalchemy's markers should not be making it into the
Pipfile.lock
, but I don't really know how to interpret PEP 508 in the context of a tool like pipenv (PEP 508 seems to be written more from the perspective of a tool like pip, where the job is to install dependencies, and it actually makes sense to evaluate environment markers, because pip knows which machine it's installing dependencies on).Either way, I'm confident that pipenv should behave deterministically: either always including those environment markers from sqlalchemy, or always not including those markers.
Steps to replicate
Workaround
This all seems to go away if I explicitly add
greenlet
as a top dependency:The text was updated successfully, but these errors were encountered: