[RFC] The next generation of virtualenv

[gaborbernat]

Following the discussion in 1362 it became apparent that for virtualenv to be able to support the new world (Windows Store installs, venv interpreters) we need some major change of direction.

Here's my proposal for going ahead with this project. This will be a major refactor for the project, that plans to keep existing interface compatibility though.

Project goal

• create a fresh copy of the system (invoker) python that has its own custom, separately controllable package list,
• an interface and behaviour that's as much as possible consistent across all supported Python environments (e.g. backport new venv features to older interpreters),
• extensible (both creation and shell activation script support wise),
• PyPi package (ability to upgrade out of band with interpreter).

Planned features

• universal pypi wheel,

• cross-platform - Windows, all UNIX flavours, MacOS.

• support for most supported pythons, the initial pool of supported Python version: python2.7, python3.4, python3.5, python3.5, pypy3, pypy2 (hoping for IronPython and Jython at some point - any other ones we should support?)

• A two year grace period of support: our interface will keep the interpreter support for two more years past its EOL: creating virtual environments is so basic that this package is the last one that should drop compatibility. During this transitional two years period, we guarantee bug fixes and compatibility. New feature support is a best effort though.

• ability to specify the target python (we'll use PEP-514/PEP-394/explicit link to discover these versions) and create virtual environments even if that target does not have this package installed

• prefer built-in venv: if the target python has venv we'll create the environment using that (and then perform subsequent operations on that to facilitate other guarantees we offer)

• ability to provision seed packages (after the creation of the virtual environments this packages will be already installed):

  • we accept PEP-508 format,
  • ability to reach out to PyPi to get latest matching the spec, or offline only (such install packages must be offline)
  • by default pip, setuptools and wheel are the seed packages (these packages are also automatically injected as offline wheel packages)

• interfaces:

  • CLI interface (python -m virtualenv, virtualenv)
  • wheel interface (env PYTHONPATH=/t/path/to/virtualenv.whl python -m virtualenv) - universal wheel,
  • zipapp interface,
  • API interface: import virtualenv; virtualenv.create("target")

• shell support - activation/deactivation scripts and prompt environment variables - by default we generate:

  • bash / zsh,
  • csh,
  • fish,
  • powershell,
  • xonsh,
  • cmd,
  • python,
  • a well-defined API to install custom shell scripts (maybe as part of the plugin system).

• three-layered configuration system, each option is determined as follows:

  • first, ini configuration support (a global ini config present in the users home),
  • second, environment variables having the VIRTUALENV_ prefix,
  • finally, command line options (argparse powered)

• plugin system these are extension points the user can inject their own custom logic, and generate an extended version of virtualenv (current bootstrapping logic):

  • extend parser (add your own custom CLI flags),
  • adjust options (change options after CLI parse),
  • after install (perform some operation once the virtual environment creation is done)

• virtualenv support - the operation should work even if the invoking python is a virtualenv created python,

• venv support - the operation should work even if the invoking python is a virtualenv created python,

• relocatable environments: an environment should keep working as long as the root python is not removed from the OS (e.g. renaming the root environment folder should not break things).

Difference from stdlib venv

How we differ from the standard library venv? The virtualenv package plans to be:

• a PyPi package, as such will be more often upgraded, easier to keep up to date and offer feature parity across pythons,
• built-in interpreter discovery and cross interpreter support,
• more interfaces (zippapp, wheel, installed package),
• easier customization - plugin system,
• be the testing ground for new features (which down the line may make it into venv),
• longer EOL.

In general, offer features that other downstream tools (for example tox, pre-commit, pipenv, pipsi, pipx) who need virtual environments as an API would want.

PyPy members (and the public please too) share your thoughts or plus one if you agree. @pfmoore @dstufft @di @pradyunsg @cjerdonek @ncoghlan @jaraco @techalchemy @uranusjr @pganssle @benoit-pierre @dholth @lkollar @takluyver @zooba

[jaraco]

An ambitious proposal!

ini configuration support (a global ini config present in the users home)

Consider using appdirs (or similar) to discover configuration in platform-preferred locations.

[gaborbernat]

appdirs well could be an option, though then we'd need to start the business of vendoring stuff into our package. To be fair 70% of this is already done, would just need a better reorganization.

[pfmoore]

Why vendor rather than just depending on it? (Not that it's hard to vendor, it's just a single file). IMO, we really need to accept that (with the notable exception of pip) packaging tools really should be built just like any other application. If having dependencies is sufficiently hard that PyPA apps avoid them, what does that say about how well we're doing making it easy for our users to develop applications?

(Sorry, not making some sort of big point about virtualenv here, I just see so many people saying "adding a PyPI dependency isn't a big deal", then the PyPA tools I work on vendoring stuff makes me wonder...).

[gaborbernat]

Without vendoring the cross interpreter feature will be hard I think. E.g. creating a virtual environment for an interpreter not having virtualenv installed.

[pradyunsg]

PYTHONPATH=/t/path/to/virtualenv.whl python -m virtualenv

Uh, this would mean that we can't depend on any additional packages from virtualenv. Since we will provide a zipapp, what usecases does this enable?

[pfmoore]

@gaborbernat You mean the -p option? I guess so. It seems like exactly the sort of situation that zipapps were intended to help with, but no-one has ever really done much to thrash out the details of making it work well.

From @pradyunsg:

Uh, this would mean that we can't depend on any additional packages from virtualenv

Actually, as @gaborbernat said, the -p option probably implies that :-( But I do wonder why we need the "wheel on PYTHONPATH" option as well as the zipapp - especially given that wheels on PYTHONPATH are unsupported in general (we use them internally within virtualenv because of the usual pip bootstrapping reasons - pip is pretty much an exception to everything ;-))

[pradyunsg]

two year grace period of support

Hmm...

My worry here is that the network effects of this might cause slowdown of adoption of newer versions of Python and the combinational explosion of number of supported Pythons. CPython is discussing different release cadence for 3.9, so 2 years might even be longer than what some releases might have from core devs.

That said, I don't want to be blocking this but mostly just making sure I voice that I'm not keen on doing this.

(hoping for IronPython and Jython at some point - any other ones we should support?)

Definitely. @gaborbernat probably knows I'd say this -- these would be good to have but I'm pretty sure folks who are more familiar with the implementation would be best positioned to do this. At the very least, in terms of support in the long term, and ideally, also the work of getting it well supported in the first place (CI, tests, etc).

first, ini configuration support (a global ini config present in the users home),

I'm biased -- I'd really like to see this be TOML instead. :)

(I have not completely processed this yet since I have to go somewhere -- but this does look like a great overall goal!)

[gaborbernat]

we already can't really depend on additional packages because of the cross interpreter feature, so I don't consider this as a major drawback, and I would like to have a simple no install/dependency way of running virtual environments (needed when people invoke us from node packages). Given it does not cost us much offering besides zipapp seems cheap.

[pganssle]

prefer built-in venv: if the target python has venv we'll create the environment using that (and then perform subsequent operations on that to facilitate other guarantees we offer)

I'm not sure I understand the rationale for this. It seems like it will make the maintenance of virtualenv harder and it will make the behaviors harder for end-users to understand. setuptools is already going the other way by trying to absorb distutils rather than continue to monkey patch it.

[gaborbernat]

@pganssle the scope is different and the problem space too. virtualenv problem is that often systems place additional constraints on system packages, so they monkey patch the built-in venv to adhere to these. Replicating all these would be hard, see e.g. #1362.

[dstufft]

FWIW I think using the built in venv package for isolation is the right idea, it's built into the actual interpreter (rather than distutils which is just a stdlib module) so it can do things much much cleaner than virtualenv can.

Virtualenv also wouldn't need to monkeypatch anything, it would just use the public API which should be fine.

FWIW I would recommend checking my old rewrite virtualenv branch, it did a lot of these things already.

[dstufft]

I would probably suggest a plugin system isn't super useful for virtualenv though, at most there's likely to just be one hook, a post create step and even then you can probably get away just having a list of things to install.

[dstufft]

ALSO, you can still depend on stuff "naturally" and still use zipapps, you just have to vendor inside the zipapp itself. I would recommend moving away from the re-execute under target python option, it is a bad thing and can be done much cleaner.

[pradyunsg]

#697 -- @dstufft's older PR