Disable upgrades to existing python modules which were not installed via pip

[nehaljwani]

What's the problem this feature will solve?

pip lets me upgrade python modules installed by the system package manager

Describe the solution you'd like

pip should not touch packages not installed by it ;-)

$ rpm -qf /usr/lib64/python2.7/site-packages/nacl
python2-pynacl-1.2.0-2.fc28.x86_64

$ pip freeze
cffi==1.11.5
ply==3.9
pycparser==2.14
PyNaCl==1.2.0
six==1.11.0

$ sudo pip install --upgrade pynacl
...
Successfully installed pycparser-2.18 pynacl-1.2.1

Alternative Solutions

🤔

Additional context

I initially posted it here. For starters, I have opened a PR to make pip_installed_by_pip() function more generic so that it can be used by whoever implements this feature.

[pradyunsg]

For starters, I have opened a PR to make pip_installed_by_pip() function more generic so that it can be used by whoever implements this feature.

I've gone ahead and merged the PR; mostly because it seems like a good improvement on it's own.

[pradyunsg]

This is an interesting idea - on the surface, ISTM that this would make pip's interactions much better when interacting with distro-installed packages and I don't notice any glaring issues here.

I'm definitely not super-familiar with the details here so I'll defer to others for inputs on this. :)

[hroncok]

Also read https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe for what we attempted in Fedora. There, however are issues currently with sudo pip install -U described in https://bugzilla.redhat.com/show_bug.cgi?id=1550368 and a for review stalled workaround at https://src.fedoraproject.org/rpms/python-pip/pull-request/10

[hroncok]

Also, from the example that seems like Python 2. We only did it for Python 3, bacause we feel like we don't want to kick a dead horse.

[dstufft]

I don't think pip should only touch files installed by pip, files installed by easy_install or any other python level package manager should be perfectly fine to modify.

What I think is the better test here, is if a package is system managed or not. We don't currently have a marker for that, but ideally we can add one and start to use that instead.

[nehaljwani]

How about adding a flag --installer to pip to mark the installer, so that one could do:

pip install <pkg-name> <extra-flags> --installer=rpm/deb/dnf/yum/conda/...

and if not specified, default to pip?

This is similar to https://www.python.org/dev/peps/pep-0376/#installer

[ncoghlan]

I agree with @dstufft that only touching files installed specifically by pip by default would be problematic, since it inhibits migrating to pip from other non-system installers.

What I do like though is the notion of being able to configure pip to be more opinionated about what it's willing to touch, and provide a reasonably helpful error message when refusing to modify things. That way a distro-provided pip can ship with that configuration set, and python3 -m pip --upgrade pip will fail with guidance on what the user should do instead.

I'm thinking along the lines of a python-installers.toml file that can appear alongside any pip.conf with entries like:

[installer.dnf]
modify = false
message = "This is a system managed package, use `dnf` to update it."

(Tangentially related: #5424 is a proposal to provide a way to add an extra "wrapper" identifier to the User Agent info that pip sends to PyPI. If that ends up being pursued, then it may make sense to structure it so that the wrapper info also ends up in the INSTALLER file)

[ChrisBarker-NOAA]

So an issue to be considered:

I don't know how yum, apt, homebrew, etc install python packages, but:

conda-build is kind of a wrapper around existing installation methods. So a conda recipe for a python package generally uses pip to install. And thus a python package installed by conda was, in fact, installed by pip, and we'd like to keep it that may.

So what would be nice is a way to tell pip whether you want it to manage the package in the future:

pip install --not_pip_managed a_package

and then, in the future, if someone did:

pip uninstall a_package

they get a "this package is not managed by pip" message.

This isn't that different than @ncoghlan's idea above, except perhaps conceptually -- the question is not whether pip was used to install it, it's whether pip is supposed to be used to manage it in the future.

[pfmoore]

Just to make a small point here - the distribution metadata format is standardised for precisely this reason - there's nothing stopping a tool like conda simply doing a pip install and then directly modifying the content of the INSTALLER file in the .dist-info directory to reflect that conda owns this distribution.

That's not to say that pip couldn't have an option to do this, just that it's perfectly acceptable to do it manually (or in this case, in conda-build).

[nehaljwani]

Instead of adding post-install hack specifically in conda-build, I would very much like to have an option which pip would provide, so that other package managers could use as well. Combining that along with @ncoghlan's proposal which would require the file to have a [installer.conda] section, and making pip honor it, would be pretty awesome.

[pfmoore]

I don't consider it a "post-install hack", but rather a supported way for installers to record their ownership of an installed package. Would it help if there was a library function (in some packaging library) that let you set the installer for a distribution? In some ways, I consider letting pip set INSTALLER to something other than "pip" to be the hack - needed by tools that want to use pip as a "subroutine", but not required for pip's core functionality.

Anyway, I'm not going to make a big deal of this, I just wanted to remind people that it's 100% possible and supported to do this without needing changes to pip.