Disallow explicitly passing install-location-related arguments in --install-options

[chrahunt]

What's the problem this feature will solve?

Currently the --install-option parameter allows passing arbitrary arguments to setup.py for packages using the legacy-installation path. If any options/flags that control install location are passed, such as:

• --exec-prefix
• --home
• --install-base
• --install-data
• --install-headers
• --install-lib
• --install-platlib
• --install-purelib
• --install-scripts
• --prefix
• --root
• --user

then it causes issues. Not every package uses setup.py, so in the general case this causes legacy packages to be installed with the the scheme provided in --install-option and PEP 517/wheel packages to be installed with the scheme pip is aware of. This has already hit two users (#7253, #7240).

Pip knows how to use scheme information properly for PEP 517, wheel, and legacy installs. Enforcing that pip is the only one passing this kind of information will make the user experience more consistent.

Describe the solution you'd like

Explicitly fail pip install if --install-option includes any of the above arguments, whether provided on the command-line or in a requirements file.

In addition to the consistent user experience, this would also allow pip to pass the more explicit

• --install-{purelib,platlib,headers,scripts,data} (for install)
• --install-dir, --script-dir (for install -e)

to setup.py instead of the current --prefix, --home, --root, --user, and --install-headers.

This may also be required to implement PEP 582 (__pypackages__) since doing legacy installs to a plain lib directory isn't possible with the current arguments being passed to setup.py, and passing an explicit --install-lib would break any usages of the other arguments in --install-option.

IMO this could be implemented without deprecation because the current behavior demonstrated in #7253 and #7240 causes failures in non-obvious ways, and the remediation is simple in most cases - just pass the parameters to pip instead.

Alternative Solutions

• do nothing - as we accumulate more features that conflict with these arguments being passed to --install-option we will get more support tickets. These will taper off eventually as people stop using these flags in order to install non-legacy packages. We eventually forget that the user can pass these options, and implement new features without worrying about it.
• automatically determine a scheme based on --install-option - this defeats the purpose of having pip-level arguments
• only fail the install if the scheme in --install-option conflicts with the one specified at the pip level - this is more friendly, but I feel that it would also be more confusing and add to the implementation burden in pip (reconciling --install-option across different packages in a requirements file)

Additional context

• pip 19.3.1 isn't respecting --install-option="--prefix=/install" for some packages #7253 - user already having a related issue
• Difference in binary builds between pip 19.3 and 19.3.1 #7240 - user already having a related issue
• Cleanup handling of installation schemes #6052 - without executing the current proposal the refactoring mentioned here gets much more difficult, since we must continue to pass --home, --prefix, --user, and --root to setup.py

[pradyunsg]

Sounds like a good idea to me.

We'd have to go through a regular deprecation cycle here but I don't see any reason to not do this.

[chrahunt]

I would argue that this is an extension of #6606/fix for #7240, and thus doesn't require a deprecation cycle. As mentioned above, the current behavior when using these flags is broken unless all builds are going through the legacy code path.

[pradyunsg]

nods

I don't think this is an extension of #6606 -- that PR changed when pip used PEP 517, which is something we're still working on. This, however, affects everyone on the legacy code path. We'd be removing functionality that user might depend on directly, so we should go through the deprecation cycle for it.

FWIW, I now know that basically any change in pip's build logic will affect someone (i.e. break their deployment / CI pipeline) and they're usually not happy about it. With the power of retrospect, I feel we should've gone for at least short deprecation cycle (1 release) for #6606 since that was a (minor) change in end-user-affecting behavior as well.

[chrahunt]

OK. After thinking on it more, we can communicate the workaround in the deprecation notice. Hopefully that means currently impacted users will not have to search the issue tracker when some packages are missing, so I don't mind that approach as much as I did initially. It might even be worth including in a fix release for 19.3? Then we could also close #7240.

[pradyunsg]

Given that we've already closed that issue (which is a form of communication on our release processes), I'm inclined to say no -- but it's really an RM's decision so... pinging @xavfernandez! :P

FWIW, I'm in no hurry to close #7240 but yes, even the deprecation messaging is enough to justify closing it.

[xavfernandez]

I'm on board with forbidding conflicting options in --install-options and also in favor of a deprecation period.
Concerning a 19.3.2 release, it seems strange to make a patch release just to include a warning... but if everyone agrees (and if the patch is clean), why not :)

[chrahunt]

Note that #7002 (unreleased) will also break some usages of these options in --install-option, since we only take into account the --user, --prefix, --target, and --root options explicitly passed to pip when determining whether to do a user install.

[chrahunt]

After talking this over with @pradyunsg, our conclusions are:

1. we should maintain the 6 month deprecation period for this (so if we make a patch release or get this in 20.0, we remove in 20.2)
2. Fix PEP 517 builds for packages without setup.py #6606 and Default to --user install in certain conditions #7002 should not be reverted, and we can treat the deprecation notice as a warning that things may not work as expected (with wheels and automatically-determined user-mode installation)
3. in the interim, we may be able to distill the options passed to --install-option into input for our scheme (potentially blocking if there is some conflict between --install-option and arguments passed directly to pip), which would let us press forward with other refactoring that requires pip to have more control over the setuptools arguments

[chrahunt]

For the more fine-grained option handling, I'm thinking the following:

install options mapped to pip options

if provided in any --install-option, these options would be compared against all other requirements' --install-options and the options provided to pip itself. If the pip equivalents are unset then they would be set to the provided options and the options removed from --install-option. If the pip equivalents are set to a different value, then we raise an error. If other requirements have a different value or do not have the same value in their --install-options then we raise an error.

1. --home maps to --target
2. --prefix maps to --prefix
3. --root maps to --root
4. --user maps to --user

install options that override scheme values directly

If provided in any --install-option, these options would be compared against all other requirements' --install-options. If the other requirements have a different value or do not have the same value in their --install-option then we raise an error. If there are any pip-level location options then we raise an error. Otherwise we explicitly set the corresponding field on our install scheme after we generate the default one and remove the argument from --install-option.

1. --install-data
2. --install-headers
3. --install-lib
4. --install-platlib
5. --install-purelib
6. --install-scripts

unsupported install options

These options, which do not have an equivalent in the scheme that we install packages to, are unsupported and if they are provided in any --install-options then we raise an error

1. --exec-prefix
2. --install-base

---

I think this is the minimum (in terms of user-facing impact) we can do while still being able to move forward on creating an explicit scheme for install.

Just to reiterate why this is important: once the above is in place, we would be able to do the following:

1. reconcile input arguments
2. map input arguments to a Scheme object in InstallCommand.run
3. use that Scheme to instantiate a working set
4. use that working set for finding installed distributions during dependency resolution (cleaning up a lot of Resolver/RequirementPreparer/InstallRequirement in the process)
5. use that scheme as the basis for installation (and conflicting requirement uninstallation), which should fix several bugs (pip install -U does not remember whether a package was installed with --user #1056, pip install --user -e tries to uninstall system-wide version, unlike pip install --user #4296, Provide a --target option on uninstall (to undo install with --target) #5595)

[sbidoul]

Added to milestone 20.2.

[pradyunsg]

Was a deprecation warning added in 19.3/20.2? Or is this an optimistic "we'll add the warning in 20.2"?

[sbidoul]

@pradyunsg There is a deprecation warning pointing to this issue, with gone_in="20.2". No feedback was reported here so I assume it's now appropriate to convert the warning into an error.

[pradyunsg]

Awesome!

[pradyunsg]

The relevant bit of code:

pip/src/pip/_internal/commands/install.py

Lines 635 to 649 in 4645ecb

	deprecated(
	reason=(
	"Location-changing options found in --install-option: {}. "
	"This configuration may cause unexpected behavior and is "
	"unsupported.".format(
	"; ".join(offenders)
	)
	),
	replacement=(
	"using pip-level options like --user, --prefix, --root, and "
	"--target"
	),
	gone_in="20.2",
	issue=7309,
	)

[chrahunt]

So, unfortunately, I didn't include --install-dir and --script-dir in the list of things to get checked. This is supported by its usage in this test, which started to fail when I refactored editable installation to explicitly pass --script-dir to setup.py. These values are only relevant to editable installs.

As a result we can't do a full replacement now like I was intending. We can still take an approach like I described in #7309 (comment), and it should be quite a bit easier since there are only two values and they map directly to the scheme values we should use (overriding "scripts" and "lib_dir"). We could do that while waiting another 2 releases before disallowing --install-dir and --script-dir and still get the majority of the benefits.

[BastienFaure]

@chrahunt I've been looking along this thread and haven't been able to find my answer, hopefuly you can give me an answer :)
I previously legeraged --install-option="--install-scripts=/custom/path/" to deploy only scripts to a specific location but this is now rejected with the recents updates.
Is there still a way through pip to install scripts to a custom location without moving the whole package ?

Thank you

[chrahunt]

Hello! Do you mind creating a new issue with this question and maybe a few more details on your use case so we can provide better advice?

[nugend]

pip documentation still gives "--install-scripts" as a suggested use of "--install-option"

[uranusjr]

@nugend Would you mind opening a new issue for the documentation fix?

[Tyil]

@chrahunt I've been looking along this thread and haven't been able to find my answer, hopefuly you can give me an answer :)
I previously legeraged --install-option="--install-scripts=/custom/path/" to deploy only scripts to a specific location but this is now rejected with the recents updates.
Is there still a way through pip to install scripts to a custom location without moving the whole package ?

Thank you

This is used by one of our repositories at work, to keep all scripts used at $dayJob in the same location, and I haven't been able to find any way to get the behaviour with any new options yet. Any suggestions on how to get this behaviour in the new situation are much appreciated.

[uranusjr]

As stated above, please open a new issue. I think this fits well into the recent configurable install scheme discussion, but it should be discussed as a feature request on its own, not buried under a closed issue.

[thorntonryan]

@chrahunt I've been looking along this thread and haven't been able to find my answer, hopefuly you can give me an answer :)
I previously legeraged --install-option="--install-scripts=/custom/path/" to deploy only scripts to a specific location but this is now rejected with the recents updates.
Is there still a way through pip to install scripts to a custom location without moving the whole package ?
Thank you

This is used by one of our repositories at work, to keep all scripts used at $dayJob in the same location, and I haven't been able to find any way to get the behaviour with any new options yet. Any suggestions on how to get this behaviour in the new situation are much appreciated.

For anyone else who happens to run across this.

We were also were using --install-option="--install-scripts=/custom/path/" in a similar way.

We were using it to package utility scripts or other in venvs and put those entry point / console scripts in a folder that was in path, but without wanting to add a bunch of python / pip or other dependencies (e.g. futurize/pasteurize) from the target package and its dependencies to path. Using --install-scripts helped us put "some" but "not all" of the things in a custom location.

With the deprecation and pending removal, the workaround we've settled on is to split the install into two steps:

-pip install sphinx --install-option="--install-scripts=/custom/path"
+pip install sphinx
+pip install sphinx --no-deps --force-reinstall --target "/custom/path" 

1. Installs the package and any dependency.
2. Re-installs the package in our desired folder, but since we gave --no-deps, won't include any other dependency.

Since we're working with venvs, for our usage, this has the desired effect.