Private packages not updated unless environment is removed #5791
Comments
matteius
added
Type: Bug 🐛
|
Could you help me verify if this is still an issue on this overhaul branch: #5793 |
|
Hi, I installed this branch of pipenv and my If I revert back to normal pipenv it works perfectly. Error output: My pipfile looks like: |
|
@joaomcarlos I just pushed a fix that solves the specific issue you reported of:
Thanks for catching that -- I think the branch is looking really good now, except there is still an issue with vcs+file:// paths that I am trying to sort out, and also some updates to the documentation around vcs specifiers since the current docs are referencing old pip line formats. Still curious of the branch solves the original issue reported though. |
|
We use this style though:
Because our libs are in our private gitlab instance. We arent using |
|
@joaomcarlos |
|
Thank you! Will give it a try. Edit: going on holiday, asked a colleague to check it out |
matteius
added
the
Status: Awaiting Update ⏳
|
Changed ref from our package to older one and updated the environment the way we used to before ( |
|
Hi, This is actually still happening. The refs get updated on the lock file but the code does not change until you --rm it |
|
@joaomcarlos here was a similar report that was confirmed fixed on the recent version: #5798 (comment) |
|
Not that great at graphics but hopefully this points out the issue. Basically I have a project that uses health check code from a lib called Added Pushed that to branch dev and pipenv updated the main project. Git shows that the .lock file of the main project got updated to the git commit ref that contains the potatoes. But when I run the project, no potatoes :/ 🗡️ 🗡️ 🗡️ and inspecting the code copy of the lib in the projects virtual environment shows that indeed, there are no potatoes. It will go hungry tonight. After
|
|
@matteius Tagging you, not sure if you saw my reply :) |
|
I saw your reply, but I cannot reproduce it, so leaving it open for now in case someone can help you. |
|
Actually, I think I see why this would be happening, https://github.com/pypa/pipenv/blob/main/pipenv/routines/install.py#L506 |
|
I opened a draft PR that I think will fix it if you can help check it out @joaomcarlos |
|
Right, we tested it and we saw the same behavior. Lets re-cap: We ran, on me and Andres machine, the following two commands: At this point, we established a baseline for pipenv. We commit a "potatoes 2" print to our package on the dev branch and run We tried a bunch of variations and none worked. We did however manage to get it updating by adding the With editable=true, we see both the change in ref and the change in the file. Does this help with debugging the issue? |
|
@matteius I am not sure if you got a notification about this, so pinging you. |
|
I saw, just don't have time at the moment--getting back to work after two weeks off 😅 |
|
@matteius all is gut :D No rush. I hope your holiday was good! |
|
@joaomcarlos I think the last release (two releases ago) had an improvement related to this, but would be helpful if you can confirm its fixed. |
|
@matteius I've encountered this bug again, so this time I did a little investigating -- I've established a workaround that should help establish a fix. I must note, that in my case, it also affects GitHub refs, such as: django-grappelli = {editable = true, ref = "9a818ed4336d190dd9464d1f26668760be077dbf", git = "https://github.com/sehmaschine/django-grappelli.git"} # Based on "==3.0.9"The same situation of not actually upgrading the package when running
So there's quite a bit I have to share, not sure whether to:
|
|
@myii some combination of 1&3 sounds good to me! |
@matteius OK, I'll share my findings here first, then I'll submit a PR depending on your feedback. TL;DR -- A quick workaround that I've been usingIn case @joaomcarlos and/or @AndreGraca98 are still affected by this bug and want to try out a workaround, without trawling through the extended discussion below. Based on the latest release (v2024.0.1): Lines 731 to 735 in 7493d18 Add the following two lines: if match is not None:
+ if (req.link and req.link.is_vcs):
+ return False
if req.specifier is not None:
return SpecifierSet(str(req.specifier)).contains(
match.version, prereleases=True
)Identifying the regressionv2023.9.8 was the last working version for me, so I used that to help narrow down my
Running interactive debugging, I could see that my VCS-based entries were entering this Lines 772 to 775 in 85c1ba5 I.e. These entries were no longer getting to the specific conditional they were supposed to reach. Hitting a traceback with the first attempted workaroundThe obvious workaround to attempt first was to bring the specific conditional back to the top. So based on the current latest commit in the Lines 746 to 750 in b99ea6b However, this led to a traceback due to the $ pipenv sync
Loading .env environment variables...
Installing dependencies from Pipfile.lock (89ef39)...
Traceback (most recent call last):
File ".../.local/bin/pipenv", line 8, in <module>
sys.exit(cli())
File ".../pipenv/vendor/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File ".../pipenv/cli/options.py", line 58, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File ".../pipenv/vendor/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File ".../pipenv/vendor/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ".../pipenv/vendor/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ".../pipenv/vendor/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File ".../pipenv/vendor/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File ".../pipenv/vendor/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File ".../pipenv/vendor/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File ".../pipenv/cli/command.py", line 651, in sync
retcode = do_sync(
File ".../pipenv/routines/install.py", line 339, in do_sync
do_init(
File ".../pipenv/routines/install.py", line 680, in do_init
do_install_dependencies(
File ".../pipenv/routines/install.py", line 438, in do_install_dependencies
batch_install(
File ".../pipenv/routines/install.py", line 503, in batch_install
deps_to_install = [
File ".../pipenv/routines/install.py", line 506, in <listcomp>
if not project.environment.is_satisfied(dep)
File ".../pipenv/environment.py", line 734, in is_satisfied
if match.has_metadata("direct_url.json") or (req.link and req.link.is_vcs):
AttributeError: 'PathDistribution' object has no attribute 'has_metadata'With a bit more digging, I found that this was something that slipped through during the Finalising the workaroundNot to be presumptuous about the correct fix for this, I expunged the clause mercilessly! This led to the fuller form of the workaround, which may be the basis of a potential fix: if match is not None:
+ if (req.link and req.link.is_vcs):
+ # VCS installs we assume are not satisfied since due to
+ # skip-lock we may be installing from Pipfile we have insufficient
+ # information to determine if a branch or ref has actually changed.
+ return False
if req.specifier is not None:
return SpecifierSet(str(req.specifier)).contains(
match.version, prereleases=True
)
if req.link is None:
return True
elif req.editable and req.link.is_file:
requested_path = req.link.file_path
if os.path.exists(requested_path):
local_path = requested_path
else:
parsed_url = urlparse(requested_path)
local_path = parsed_url.path
return requested_path and os.path.samefile(local_path, match.location)
- elif match.has_metadata("direct_url.json") or (req.link and req.link.is_vcs):
- # Direct URL installs and VCS installs we assume are not satisfied
- # since due to skip-lock we may be installing from Pipfile we have insufficient
- # information to determine if a branch or ref has actually changed.
- return False
return True
return False |
Analysis of Pipenv Issue #57911. Problem SummaryThe core issue is that 2. Comments Analysis
3. Proposed ResolutionThe root cause lies in the Code ChangesFile: Function: Current Code (snippet): if match is not None:
if (req.link and req.link.is_vcs):
# VCS installs we assume are not satisfied since due to
# skip-lock we may be installing from Pipfile we have insufficient
# information to determine if a branch or ref has actually changed.
return False
# ... (other checks) ...Proposed Code (snippet): if match is not None:
if (req.link and req.link.is_vcs):
# Compare the commit IDs from the requirement and the installed distribution
if req.req.vcs_info.commit_id != match.direct_url.info.commit_id:
return False
# ... (other checks) ...This change compares the commit IDs from the 4. Code SnippetThe proposed code snippet is already included in the resolution section (point 3). 5. Additional Steps
By implementing these changes, Pipenv can ensure the correct and consistent updating of VCS dependencies based on the latest commit ID specified in the |
|
Could this be re-checked against #6276 which has some bug fixes to |
Issue description
Private packages defined with an object like
mylib = {ref = "dev" git = "ssh://git.mycompany.com/my-lib.git"}are not being updated on my virtual env, even after the pipenv update creates a new Pipfile.lock with the correct commit id.Expected result
After calling
pipenv update, I expect the Pipfile.lock ref to change and also the code in my environment to change, such that I have access to the new code within my project.Actual result
I see the Pipfile.lock ref change, but the actual code for my dependency does not change.
If I do
pipenv --rmto remove the environment and then re-create the environment, I can get the actual code change to replicate on disk.Steps to replicate
Update
my-libwith some code change, commit and push. On my project, dopipenv updateand wait for the process to finish.Check that the
Pipfile.lockhas changed and has the correct ref for the new commit inmy-lib.Check that the code for
my-libhasnt actually changed within my original projects virtual env.The text was updated successfully, but these errors were encountered: