Feature: incorporate pip-run as pip run

[jaraco]

In an effort to obviate easy_install (including implicit invocations such as setup_requires directives), I've created rwt. If you're not familiar with rwt, take a look at that readme, which describes its purpose and gives some examples.

I've been using rwt in my daily workflows and I'm finding it quite powerful as a bootstrapping tool. However, it suffers from the same bootstrapping problem as setuptools and pip and virtualenv -- you have to install it before it's available to use it, so breaks the 'one step' workflows and adds a step to the instructions you give to someone (install python, install pip, install rwt, run this command).

I'd like to consider bundling rwt with pip as the pip run command. Since rwt is mostly a thin wrapper around pip and its parameters are passed directly to pip install, it makes sense as a pip command, and adding it as a command to pip would solve the bootstrapping issue. The functionality would also then be available during pip install, potentially eliminating the setup_requires reliance on easy_install (or at least supporting an easy_install-free option).

So I'm proposing two things:

1. Bundle rwt as pip run and make it the pypa-endorsed mechanism for on-demand dependency resolution.
2. In pip install when building a source dist, invoke setup.py using rwt.

In the short term, the pip run command can be used for bootstrapping or test runs or whatever ad-hoc requirements one might have.

> pip run requests
...
>>> import requests
>>> requests.get(...)

discarded approach

Once this functionality has established more thoroughly and packagers can rely on all pip versions having this functionality, they could eliminate the reliance on distutils altogether and use whatever dependencies they require to build their package. Consider the pbr use case. Instead of invoking setuptools.setup() in their setup.py, they do this:

__requires__ = ['pbr']

import pbr
pbr.setup()

Where pbr might have dependencies on setuptools or distutils and and invoke that behavior, or perhaps they use another build system. Additionally, pip might be able to eschew the responsibility of supplying setuptools. If a script wants to rely on setuptools, it can require that:

__requires__ = ['setuptools']

import setuptools
setuptools.setup(...)

That last bit (pip without setuptools bundled) makes my heart glow a little. Think it over, maybe try out rwt, and let me know what you think. I plan to put together a PR after addressing any concerns.

[RonnyPfannschmidt]

@jaraco broken link

[dholth]

That's beautiful. I've long suspected that environments based on "what a particular console script requires" might be a great alternative to environments based on "paths relative to a copy of the Python executable", but you went and did it!

[dstufft]

So I don't have a real strong feeling about this. It doesn't feel like something I'd ever use personally, but I already have virtualenvwrapper so when I want a temporary environment I just do mktmpenv and whenever I deactivate or exit my shell that environment goes away automatically without needing some sort of runtime munging. This seems to generally work better to me than what something like pip run would because it's not unusual that I do something like:

$ pip install requests
$ python
>>> import requests
>>>  # Realize I forgot I needed cachecontrol
$ pip install cachecontrol
>>> import requests, cachecontrol

Or something like that. With explicit temporary environment creation/deletion (via deactivate/exit) that is trivial to do. With rwt it seems like I'd have to go and edit my pip run command to do pip run requests cachecontrol and get a whole new environment again. It also works for things where I want a temporary script, since it sets my current directory to the temporary virtual environment so I can use that area as a scratch pad that gets automatically cleaned up.

It's entirely possible (likely even) that there's some workflow or use case that I don't personally use that would make this useful though.

However, I don't think it's something we'd want to recommend as a replacement for setup_requires. Obviously people can do whatever they want in their setup.py (which is how setup_requires works at all today). We already have PEP 518 to use as an alternative to setup_requires (it's just not implemented yet). Looking at the source code for rwt it appears __requires__ must be a static list (which makes sense, need to be able to fetch it without executing) but I think the UX of something that must be static being inside of a setup.py is not great. People are going to expect to be able to do normal Python things with it since it's Python syntax and that's going to fail.

So I guess overall my suggestion about how to implement the feature would be to ditch the Python syntax and do something like:

#!/usr/bin/python
# -*- coding: utf-8 -*-
# Requires: setuptools>=1.0
# Requires: cffi>=1.0

import setuptools
setuptools.setup(...)

I think that makes it more clear that this is not Python syntax and you can't expect to do things like string interpolation or what have you inside of it.

Given all of that though, I am -1 on recommending/using it for a setup_requires replacement and -0 on including the pip run feature, only because I struggle to think of a case where I'd personally use it.

[dholth]

This might be a great way to implement PEP 518. Surely all the interesting stuff is unrelated to how the dependencies are read. Recall that you would parse a .toml and look for the ['build-system']['requires'] key.

It looks like it would probably run into the "pip --target does not work on global pip in Debian" problem caused by their "--user is the default" patch.

I dread using programs that are written in Python even though I love developing programs in Python, and it's because of virtualenv. Virtualenv is for development. When you just want to run youtube-dl there is no python; ImportError; pip install cycle.

[jaraco]

I chose to use __requires__ because that's also the mechanism that pkg_resources uses to enforce that a given script meets its requirements. Of course, pkg_resources doesn't require that it be a static list, but I'd be more than willing to document clearly that limitation (which I do consider a feature, disallowing variation).

ditch the Python syntax and do something like

Ugh. I don't want to invent yet another DSL for package management, especially one that mixes Mime headers and shell comments and requirement syntax. I want the declarations to be data and in a widely-accepted format (i.e. Python or JSON).

We already have PEP 518 to use as an alternative to setup_requires

I do recall having read PEP 518, but PEP 518 falls short of the goals of rwt and has a narrower focus. rwt seeks to solve a more general problem. And in fact, rwt may be compatible with the implementation for PEP 518. If pip can read the PEP 518 metadata file, rwt could still be used to create the temporary context in which those packages are made available with something like:

rwt --with-pep518-build dist-info/metadata.json -- setup.py install

Not that you would necessarily want to do it that way, but you could.

I struggle to think of a case where I'd personally use it

As seasoned developers, we're comfortable with tooling and don't mind having and maintaining environments with pip and setuptools and virtualenv and virtualenvwrapper and buildout and pbr and ... But when I'm working with a less experienced Python developer (maybe a developer on a C project, or maybe a project manager, or maybe a bug tracker where I don't know who might want to run it), I'd like a supported mechanism that gives them the ability to rapidly run a script or project with minimal fuss. I'd like to be able to give them a single command to run and have it work in a large number of environments. If I give instructions on installing dependencies, I have to refer them to documentation and they have to learn the caveats (access controls, system versus user environments, virtual environments including activation, and cleanup). This technique instead trades all of that complexity with a concise invocation. All they need is a minimally viable Python environment (with pip).

It's also important that this mechanism supports a single file, so that a script can be represented in a Gist or as downloadable script.

Consider this example where I've replaced setup_requires and tests_require and pytest-runner with test requirements nicely defined and encapsulated, but separate from the project metadata. Of course, since Travis runs are themselves ephemeral, those features are more valuable for someone using the test as a recipe for running tests locally (without having to mimic Travis-CI).

That said, I'm okay with it not being something you would want to use. If you prefer for the state to accumulate in virtualenvs on your machine and virtualenvwrapper helps facilitate that, I'm all in favor of continuing to use those. I don't see rwt as ever obviating those use cases.

I am -1 on recommending/using it for a setup_requires replacement.

That's understandable. And I'm happy to defer that decision to the preferences of the pip maintainers. I present the idea here for consideration.

[jaraco]

It looks like it would probably run into the "pip --target does not work on global pip in Debian" problem caused by their "--user is the default" patch.

Yes, it does. I've found that I've had to manually install pip on Xenial systems to bypass the issue.

[pfmoore]

Having thought about it, I like the idea as a standalone utility (which I might use occasionally, but see below) but I'm not sure it fits that well as a pip subcommand. It sort of feels more like it's related to virtualenv than to pip.

Personally, I have a ve powershell function that wraps virtualenv. When I want to experiment, I typically do ve -temp and then pip install what I need (basically my own equivalent of virtualenvwrapper). I keep thinking about adding an argument to ve to auto-install stuff, just to remove one step. But either way it's easy enough to do what I want. But point taken that this is from the POV of someone comfortable with command line tools.

Using rwt to run a script and auto-manage its dependencies is quite a nice idea. I'd probably only use it for "one off" stuff, though - for scripts I wanted to distribute I'd likely use zipapp and bundle dependencies in.

[dstufft]

To be clear, other than being -1 on using/recommending this for a setup_requires replacement (i don't include using the isolation mechanism but pulling data from pyproject.toml as "using" in this context since it'd just be an implementation detail at that point) I don't feel super strongly about the feature itself in either a positive or negative way. If folks think they'd find it generally useful I wouldn't personally be opposed to it, I am not great at "stepping outside" my own knowledge and I don't do much with beginners. Perhaps it'd make sense to poke some of the people who tend to do a lot of beginner stuff and see what their reactions to this idea are?

I don't want to invent yet another DSL for package management, especially one that mixes Mime headers and shell comments and requirement syntax. I want the declarations to be data and in a widely-accepted format (i.e. Python or JSON).

So my biggest problem with __requires__ = [...] is that I feel like being a restricted set of Python, but within a regular Python file makes it an attractive nuisance. People are going to expect to be able to dynamically compute this unless they happen to have read and understood the documentation about it. You can't document your way out of a usability problem and I feel like this restricted-python-within-python is just that. That pkg_resources uses it for it's runtime stuff sort of makes this worse because it also invalidates the expectation that anyone who has used that feature has, that you can do dynamic things in it. I don't really care what the exact syntax is, but putting it in a comment completely removes people's ability to metaprogram their dependency declarations and removes that particular footgun (unless they do something even crazier like adding an encoding that does magical interpolation or something... but at that point you're well off the beaten path enough to know you're doing something unsupported).

That said, I'm okay with it not being something you would want to use. If you prefer for the state to accumulate in virtualenvs on your machine and virtualenvwrapper helps facilitate that, I'm all in favor of continuing to use those. I don't see rwt as ever obviating those use cases.

Sure! I don't think that every thing that exists in pip needs to be something I'd personally use either :). My "I don't think I'd use this" is mostly about me not having a good frame of reference to judge this particular feature by and whether it's useful to include or not. My only real concern about adding a pip run command is whether or not it passes the "will this be useful to enough people to justify the mental and maintenance overhead of another command". The downloads for rwt are fairly small:

Installer	Downloads
bandersnatch	3126
requests	65
Browser	48
Unknown	29
pip	17
z3c.pypimirror	7

Though as you mentioned, this does have the same bootstrapping problem that pip/setuptools/virtualenv has and there is also a certainly a discoverability problem too. Perhaps rebranding as pip-run (or keep it as rwt even) and linking to it from the pip/virtualenv/setuptools/packaging.python.org docs and seeing if it gains traction before adding it to pip itself might be a reasonable way to go about it? Or maybe just soliciting more feedback from groups who might find it useful? I dunno!

[dholth]

I feel strongly about features like this. I think integrating this and/or something more like pipsi into pip would be transformative - that we would see more Python application development in general.

Consider the humble npm install -g useful_program. The only way to install things with npm means useful_program is never broken just because you installed a second program, and as a result npm is a popular way to distribute command line tools that are not primarily libraries. Of course with Python the isolation mechanism would be different and we would not recursively isolate dependencies of dependencies.

Why not distribute a .pyz in the downloads section of your Wordpress blog instead? Sometimes that works, but pypi is awesome.

We do love our interactive shells in Python. In a system like rwt you would start with the script and say "give me a shell with that script's dependencies". Backwards compared to virtualenv where you start with the dependencies and eventually get the script. Which direction is best depends on what you are doing.

Also, if we build rwt into pip, maybe --target will start working on Debian again.

I'm not worried about __requires__ = [a for a in range(10)] shenanigans, rwt should be able to provide immediate and clear error messages if someone tried it.

I assume @jaraco is familiar with my own attempt at short bootstrap code https://bitbucket.org/dholth/enscons/src/tip/setup.py?fileviewer=file-view-default

[dholth]

By the way does rwt handle cases like setuptools 27 is installed, but requires = [ 'setuptools==26']

[jaraco]

Yup

$ python -m easy_install --version
setuptools 27.2.0 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (Python 3.6)
$ python -m rwt setuptools==26.0.0 -- -m easy_install --version
Loading requirements using setuptools==26.0.0
setuptools 26.0.0 from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-bn3_nm11 (Python 3.6)

[dstufft]

I don't think this feature is related to pipsi (though I also think that something like pipsi inside of pip itself would be great). This seems to be more tailored towards one-off implicit, temporary environments whereas pipsi is tailored more towards per-command, permanent environments.

I do think there is something here we need to figure out though for any potentional pipsi, rwt, and PEP 518 implementation is how is pip going to handle isolation. Currently pip and virtualenv do not have a dependency on each other (and that was done on purpose to allow people to use one without the other). However all of these things need some mechanism of an isolated install so it starts to become attractive to bolt the two things together more -- that or we need something like what rwt is currently doing, which is sort of emphereal environments based on PYTHONPATH. I think this sort of thing is also the reason why @pfmoore feels that it's more related to virtualenv than pip. If we were minimum of 3.3+ we could pretty reasonably just depend on the venv module and be done with it, but 2.7 is still a thorn there (we're dropping 2.6 in the future so that's OK).

I don't think that pip run is going to affect what Debian does with --target.

Anyways, my opinion on pip run is what's above :)

[pfmoore]

Agreed - for me the interesting thing about rwt is the isolation aspect, and that feels more like virtualenv/venv than pip.

I would be very interested in a discussion on how rwt does isolation (I haven't even looked into it myself, yet[1]) and whether that's appropriate as a standard method for pip. IMO, it may even be worth considering an informational PEP "standard approach to providing script-specific dependencies". That would be useful for people writing zipapps, as well.

[1] It may be that it's nothing more clever than "create a temp dir, do pip install -t, set sys.path appropriately, clean up afterwards".