pip install --user should check that the user's $PATH is correct

[njsmith]

On Unix, pip install --user ... drops scripts into ~/.local/bin. Historically, this directory has not been on the default Debian/Ubuntu $PATH.

This is hopefully on its way to being fixed (bash 4.3-15 is on its way into Debian unstable now, and should hopefully land in Debian testing in a few weeks, bug report here; the Ubuntu bug that will hopefully be used to justify a backport of this fix into Xenial is here). But, even once the fix lands, all it will do is change the default for newly created accounts -- so for a long time, there are going to be people who try running pip install --user and end up with it dropping executables into a directory that's not on $PATH.

When pip install --user installs a script into ~/.local/bin (or the equivalent on other OSes, why not), then it should check the current os.environ["PATH"], and if this directory is not on the PATH, then it should print an explanatory message that warns the user and gives them some information on how to fix the problem.

(Or heck, it could even suggest they run pip fix-user-path to have pip automatically add tihs to their ~/.profile -- it's a little wonky but not too hard to make reasonably robust, and if it's a manual action that tells the user what it's doing as it runs then it should be reasonable helpful.)

(This would be a reasonable first fix for someone new to pip -- does pip's issue tracker have a tag for those?)

[dstufft]

I think this is a good idea, although my one worry is that some RC files exclude directories that don't already exist so that's a bit of a weird case where it would be in their $PATH if they opened a new shell but it's not now.

I'm sort of hovering between -0 and -1 on something like pip fix-user-path just because I find those automatic profile things to be error prone.

[njsmith]

I guess I'm currently +0 on pip fix-user-path on the following rationale:

• For users who understand how ~/.profile and friends work, then it's a small convenience that they'll use or ignore as they prefer, and if it breaks things then they'll be able to fix it.
• For users who don't understand how ~/.profile and friends work, then the best they can hope to do is to blindly follow some complicated instructions, and if those instructions have an error then they're doomed regardless. So we might as well implement those instructions in the form of code, to at least eliminate human error and human pain.

For concreteness, I'm imagining a user interaction that would look something like

$ pip install --user virtualenv
Collecting virtualenv
  [...]
Successfully installed virtualenv-15.0.2

WARNING: this package provides a script called "virtualenv", and I put this
into your personal script directory, /home/njs/.local/bin/
BUT, your command line is not set up to look for scripts in this directory!
You won't be able to run the "virtualenv" script until you fix this.
For more information, see

   https://...

or, if you want me to try fixing your configuration for you automatically, run:

   pip fix-user-path

(You only have to do this once.)
$ pip fix-user-path
It looks like you're using the "bash" shell! Great, I know how to handle that.
I've added the following lines to the file /home/njs/.profile:

    # -- start lines automatically added by pip fix-user-path --
    export PATH=$HOME/.local/bin:$PATH
    # -- end lines automatically added by pip fix-user-path --

If this looks wrong, you can edit that file directly in your favorite text editor.

PLEASE NOTE: You probably need to log out and log back in before the
new configuration will take effect!
$ pip fix-user-path
It looks like you're using the "bash" shell! Great, I know how to handle that.
...I just checked the file /home/njs/.profile, and it looks like I've already fixed
your configuration for you, but it hasn't taken effect yet.
Most likely, this means you need to log out and log back in again.
If that doesn't help, [...]
$ logout
<start new shell>
$ pip fix-user-path
It looks like /home/njs/.local/bin is already on your $PATH, so there's
nothing for me to do -- you're all set!

[njsmith]

some RC files exclude directories that don't already exist

Oh, and on this point:

Fedora/RH unconditionally add ~/.local/bin to $PATH:

Debian/Ubuntu have historically done the annoying thing you mention with ~/bin, but that's irrelevant to us; the new patch that's working its way through the system to add ~/.local/bin to the default $PATH configuration adds it unconditionally (and actually makes ~/bin unconditional as well): https://sources.debian.net/src/bash/4.3-15/debian/skel.profile/

Is there anyone else you're worried about in particular?

[Ivoz]

I've seen plenty of other 3rd party python distributions maul people's profile scripts adding what they saw fit to paths which then made them a nightmare when later "uninstalled" or someone used something else. Definitely -1 on messing with that, especially given pip has X many linux distros to all work on as well as macOS.

[njsmith]

@Ivoz: the difference is that (a) this path is always supposed to be there, there's no sense in which a user might "uninstall" their ~/.local directory, and all we'd be doing is fixing existing user accounts to match how they were supposed to be set up all along, (b) pip by definition knows what the standard user script directory is on all linux distros and macOS, so there's no problem there, and (c) this is something that users would explicitly opt into.

"Don't mess with user profiles" isn't an option -- unless we want to disable --user entirely. The options are "force users to mess with their profiles by manually cutting-and-pasting things" or "provide a tool that does the cutting-and-pasting for them".

[dstufft]

So here's a question, it seems to me that we can get most of the way there by just having a suggested snippet in the error message without any sort of fix-me-up command. Yes it'll require them to copy/paste but I think it's a fairly trivial operation for them to do that, and we avoid any sort of "pip broke my profile" issues.

[dstufft]

Er, my question being, why not just do that?

[njsmith]

That would also be a lot better than what we have now :-).

(Do keep in mind though that a large chunk of pip's userbase does not know what a "shell" is, or what a "dot-file" is, or even what a "text editor" is ("you mean like Google Docs?")... trivial is a relative term :-))

[jayvdb]

A cross-platform bash source-able fix-me-up would be useful for CI scripts
e.g.

$ source <(pip fix-me-up-source)

e.g. travis-ci/travis-ci#6247 could use something like that, so they dont need to hard-code the path names.

And it could be the basis for users automating the .profile fix themselves.

$ pip fix-me-up-source >> ~/.profile

Alternatively, is there some way to to ask pip to emit the relevant directories it will use, in a machine readable format, sort of like what pkg-config does for C libraries.

[njsmith]

For CI scripts you probably just want to use virtualenvs or something instead of --user? Though yeah, it sucks if they just randomly allow --user but it's broken.

Alternatively, is there some way to to ask pip to emit the relevant directories it will use, in a machine readable format, sort of like what pkg-config does for C libraries.

Maybe that should be a different bug, since this one is aimed at trying to fix a papercut that bites new users? (While you're at it, a similar thing I've been annoyed by is that there's no way to ask pip to say what wheels it thinks can be installed in the current environment.)

[hjwp]

for info, I think there is a similar problem on windows? I just tried on win 10 VM, and pip install --user put my script in ~/AppData/Roaming/Python/Scripts/, which is not on the path (either in dos or bash)

[pradyunsg]

Cross linking to #3813 because it has a dependency on this.

[piotr-dobrogost]

@pradyunsg
Something must have gone wrong as you linked this issue to itself above.

[pradyunsg]

Well, yes. I posted this on the wrong one. I'm sleepy.

[pradyunsg]

Cross linking to #1668 because it has a dependency on this.

[pradyunsg]

(This would be a reasonable first fix for someone new to pip -- does pip's issue tracker have a tag for those?)

It does. I'll tag it. :)

[ofek]

I just released https://github.com/ofek/pybin to ameliorate this issue. Please let me know how this works for anyone that tries it!

[pradyunsg]

Thoughts on adding a pip self-check command that checks this and maybe a bunch of other things? I dunno what the other things would be; I like the spelling. If we don't have some other stuff pip could check a different spelling is fine too.

[dstufft]

Personally I think the right way is to do this check during the install and make it a warning. An additional command someone has to remember to call seems like most folks would just forget to do it.

[ncoghlan]

I agree with Donald - the most useful place for a warning would be during installations where the package actually puts something in the sysconfig scripts directory (https://docs.python.org/3/library/sysconfig.html#installation-paths)

I'm not sure of the best place to hook this in to pip's processing, but one way to go would be to call shutil.which() after installing scripts if stdin and standard out are tty's and complain if the just-insatlled script can't be found (the latter restriction would be to avoid false alarms during something like RPM's "install" step - that's not a real installation, it's an installation into the buildroot).

[pradyunsg]

This is the response I was expecting. Just wanted to make sure of it before moving forward. I have an open PR for the warning mechanism with wheels. I'll rebase it or merge the current master into it when I get in front of a bigger screen, in a few hours. I'm still unsure about what the warning should say though... As always, any inputs on that PR are more than welcome. :)