Implemented PEP 405 (Python virtual environments).

Author: vsajip

Doc/library/development.rst

@@ -23,3 +23,4 @@ The list of modules described in this chapter is:
    unittest.mock-examples.rst
    2to3.rst
    test.rst
+   venv.rst

Doc/library/sys.rst

@@ -29,6 +29,26 @@ always available.
    command line, see the :mod:`fileinput` module.
 
 
+.. data:: base_exec_prefix
+
+   Set during Python startup, before ``site.py`` is run, to the same value as
+   :data:`exec_prefix`. If not running in a virtual environment, the values
+   will stay the same; if ``site.py`` finds that a virtual environment is in
+   use, the values of :data:`prefix` and :data:`exec_prefix` will be changed to
+   point to the virtual environment, whereas :data:`base_prefix` and
+   :data:`base_exec_prefix` will remain pointing to the base Python
+   installation (the one which the virtual environment was created from).
+
+.. data:: base_prefix
+
+   Set during Python startup, before ``site.py`` is run, to the same value as
+   :data:`prefix`. If not running in a virtual environment, the values
+   will stay the same; if ``site.py`` finds that a virtual environment is in
+   use, the values of :data:`prefix` and :data:`exec_prefix` will be changed to
+   point to the virtual environment, whereas :data:`base_prefix` and
+   :data:`base_exec_prefix` will remain pointing to the base Python
+   installation (the one which the virtual environment was created from).
+
 .. data:: byteorder
 
    An indicator of the native byte order.  This will have the value ``'big'`` on
@@ -199,6 +219,10 @@ always available.
    installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y*
    is the version number of Python, for example ``3.2``.
 
+   .. note:: If a virtual environment is in effect, this value will be changed
+      in ``site.py`` to point to the virtual environment. The value for the
+      Python installation will still be available, via :data:`base_exec_prefix`.
+
 
 .. data:: executable
 
@@ -775,6 +799,10 @@ always available.
    stored in :file:`{prefix}/include/python{X.Y}`, where *X.Y* is the version
    number of Python, for example ``3.2``.
 
+   .. note:: If a virtual environment is in effect, this value will be changed
+      in ``site.py`` to point to the virtual environment. The value for the
+      Python installation will still be available, via :data:`base_prefix`.
+
 
 .. data:: ps1
           ps2

Doc/library/venv.rst

@@ -0,0 +1,193 @@
+:mod:`venv` --- Creation of virtual environments
+================================================
+
+.. module:: venv
+   :synopsis: Creation of virtual environments.
+.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
+.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
+
+
+.. index:: pair: Environments; virtual
+
+.. versionadded:: 3.3
+
+**Source code:** :source:`Lib/venv.py`
+
+--------------
+
+The :mod:`venv` module provides support for creating lightweight
+"virtual environments" with their own site directories, optionally
+isolated from system site directories.  Each virtual environment has
+its own Python binary (allowing creation of environments with various
+Python versions) and can have its own independent set of installed
+Python packages in its site directories.
+
+Creating virtual environments
+-----------------------------
+
+Creation of virtual environments is simplest executing the ``pyvenv``
+script::
+
+    pyvenv /path/to/new/virtual/environment
+
+Running this command creates the target directory (creating any parent
+directories that don't exist already) and places a ``pyvenv.cfg`` file
+in it with a ``home`` key pointing to the Python installation the
+command was run from.  It also creates a ``bin`` (or ``Scripts`` on
+Windows) subdirectory containing a copy of the ``python`` binary (or
+binaries, in the case of Windows) and the ``pysetup3`` script (to
+facilitate easy installation of packages from PyPI into the new virtualenv).
+It also creates an (initially empty) ``lib/pythonX.Y/site-packages``
+subdirectory (on Windows, this is ``Lib\site-packages``).
+
+.. highlight:: none
+
+On Windows, you may have to invoke the ``pyvenv`` script as follows, if you
+don't have the relevant PATH and PATHEXT settings::
+
+    c:\Temp>c:\Python33\python c:\Python33\Tools\Scripts\pyvenv.py myenv
+
+or equivalently::
+
+    c:\Temp>c:\Python33\python -m venv myenv
+
+The command, if run with ``-h``, will show the available options::
+
+    usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear]
+                  ENV_DIR [ENV_DIR ...]
+
+    Creates virtual Python environments in one or more target directories.
+
+    positional arguments:
+      ENV_DIR             A directory to create the environment in.
+
+    optional arguments:
+      -h, --help             show this help message and exit
+      --system-site-packages Give access to the global site-packages dir to the
+                             virtual environment.
+      --symlink              Attempt to symlink rather than copy.
+      --clear                Delete the environment directory if it already exists.
+                             If not specified and the directory exists, an error is
+                             raised.
+
+
+If the target directory already exists an error will be raised, unless
+the ``--clear`` option was provided, in which case the target
+directory will be deleted and virtual environment creation will
+proceed as usual.
+
+The created ``pyvenv.cfg`` file also includes the
+``include-system-site-packages`` key, set to ``true`` if ``venv`` is
+run with the ``--system-site-packages`` option, ``false`` otherwise.
+
+Multiple paths can be given to ``pyvenv``, in which case an identical
+virtualenv will be created, according to the given options, at each
+provided path.
+
+
+API
+---
+
+The high-level method described above makes use of a simple API which provides
+mechanisms for third-party virtual environment creators to customize
+environment creation according to their needs.
+
+The :class:`EnvBuilder` class accepts the following keyword arguments on
+instantiation:
+
+   * ``system_site_packages`` - A Boolean value indicating that the
+     system Python site-packages should be available to the
+     environment (defaults to ``False``).
+
+   * ``clear`` - A Boolean value which, if True, will delete any
+     existing target directory instead of raising an exception
+     (defaults to ``False``).
+
+   * ``symlinks`` - A Boolean value indicating whether to attempt
+     to symlink the Python binary (and any necessary DLLs or other
+     binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to
+     ``True`` on Linux and Unix systems, but ``False`` on Windows and
+     Mac OS X.
+
+The returned env-builder is an object which has a method, ``create``,
+which takes as required argument the path (absolute or relative to the current
+directory) of the target directory which is to contain the virtual environment.
+The ``create`` method will either create the environment in the specified
+directory, or raise an appropriate exception.
+
+Creators of third-party virtual environment tools will be free to use
+the provided ``EnvBuilder`` class as a base class.
+
+.. highlight:: python
+
+The ``venv`` module will also provide a module-level function as a
+convenience::
+
+    def create(env_dir,
+               system_site_packages=False, clear=False, symlinks=False):
+        builder = EnvBuilder(
+            system_site_packages=system_site_packages,
+            clear=clear,
+            symlinks=symlinks)
+        builder.create(env_dir)
+
+The ``create`` method of the ``EnvBuilder`` class illustrates the
+hooks available for subclass customization::
+
+    def create(self, env_dir):
+        """
+        Create a virtualized Python environment in a directory.
+
+        :param env_dir: The target directory to create an environment in.
+
+        """
+        env_dir = os.path.abspath(env_dir)
+        context = self.create_directories(env_dir)
+        self.create_configuration(context)
+        self.setup_python(context)
+        self.setup_scripts(context)
+        self.post_setup(context)
+
+Each of the methods ``create_directories``, ``create_configuration``,
+``setup_python``, ``setup_scripts`` and ``post_setup`` can be
+overridden.  The functions of these methods are:
+
+   * ``create_directories`` - creates the environment directory and
+     all necessary directories, and returns a context object. This is
+     just a holder for attributes (such as paths), for use by the
+     other methods.
+
+   * ``create_configuration`` - creates the ``pyvenv.cfg``
+     configuration file in the environment.
+
+   * ``setup_python`` - creates a copy of the Python executable (and,
+     under Windows, DLLs) in the environment.
+
+   * ``setup_scripts`` - Installs activation scripts appropriate to the
+     platform into the virtual environment.
+
+   * ``post_setup`` - A placeholder method which can be overridden
+     in third party implementations to pre-install packages in the
+     virtual environment or perform other post-creation steps.
+
+In addition, ``EnvBuilder`` provides an ``install_scripts`` utility
+method that can be called from ``setup_scripts`` or ``post_setup`` in
+subclasses to assist in installing custom scripts into the virtual
+environment. The method accepts as arguments the context object (see
+above) and a path to a directory. The directory should contain
+subdirectories "common", "posix", "nt", each containing scripts
+destined for the bin directory in the environment. The contents of
+"common" and the directory corresponding to ``os.name`` are copied
+after some text replacement of placeholders:
+
+* ``__VENV_DIR__`` is replaced with the absolute path of the
+  environment directory.
+
+* ``__VENV_NAME__`` is replaced with the environment name (final path
+  segment of environment directory).
+
+* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
+  (either ``bin`` or ``Scripts``).
+
+* ``__VENV_PYTHON__`` is replaced with the absolute path of the
+  environment's executable.

Lib/distutils/sysconfig.py

@@ -18,6 +18,8 @@
 # These are needed in a couple of spots, so just compute them once.
 PREFIX = os.path.normpath(sys.prefix)
 EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
+BASE_PREFIX = os.path.normpath(sys.base_prefix)
+BASE_EXEC_PREFIX = os.path.normpath(sys.base_exec_prefix)
 
 # Path to the base directory of the project. On Windows the binary may
 # live in project/PCBuild9.  If we're dealing with an x64 Windows build,
@@ -39,11 +41,18 @@
 # different (hard-wired) directories.
 # Setup.local is available for Makefile builds including VPATH builds,
 # Setup.dist is available on Windows
-def _python_build():
+def _is_python_source_dir(d):
     for fn in ("Setup.dist", "Setup.local"):
-        if os.path.isfile(os.path.join(project_base, "Modules", fn)):
+        if os.path.isfile(os.path.join(d, "Modules", fn)):
             return True
     return False
+_sys_home = getattr(sys, '_home', None)
+if _sys_home and os.name == 'nt' and _sys_home.lower().endswith('pcbuild'):
+    _sys_home = os.path.dirname(_sys_home)
+def _python_build():
+    if _sys_home:
+        return _is_python_source_dir(_sys_home)
+    return _is_python_source_dir(project_base)
 python_build = _python_build()
 
 # Calculate the build qualifier flags if they are defined.  Adding the flags
@@ -74,23 +83,24 @@ def get_python_inc(plat_specific=0, prefix=None):
     otherwise, this is the path to platform-specific header files
     (namely pyconfig.h).
 
-    If 'prefix' is supplied, use it instead of sys.prefix or
-    sys.exec_prefix -- i.e., ignore 'plat_specific'.
+    If 'prefix' is supplied, use it instead of sys.base_prefix or
+    sys.base_exec_prefix -- i.e., ignore 'plat_specific'.
     """
     if prefix is None:
-        prefix = plat_specific and EXEC_PREFIX or PREFIX
+        prefix = plat_specific and BASE_EXEC_PREFIX or BASE_PREFIX
     if os.name == "posix":
         if python_build:
             # Assume the executable is in the build directory.  The
             # pyconfig.h file should be in the same directory.  Since
             # the build directory may not be the source directory, we
             # must use "srcdir" from the makefile to find the "Include"
             # directory.
-            base = os.path.dirname(os.path.abspath(sys.executable))
+            base = _sys_home or os.path.dirname(os.path.abspath(sys.executable))
             if plat_specific:
                 return base
             else:
-                incdir = os.path.join(get_config_var('srcdir'), 'Include')
+                incdir = os.path.join(_sys_home or get_config_var('srcdir'),
+                                      'Include')
                 return os.path.normpath(incdir)
         python_dir = 'python' + get_python_version() + build_flags
         return os.path.join(prefix, "include", python_dir)
@@ -115,11 +125,14 @@ def get_python_lib(plat_specific=0, standard_lib=0, prefix=None):
     containing standard Python library modules; otherwise, return the
     directory for site-specific modules.
 
-    If 'prefix' is supplied, use it instead of sys.prefix or
-    sys.exec_prefix -- i.e., ignore 'plat_specific'.
+    If 'prefix' is supplied, use it instead of sys.base_prefix or
+    sys.base_exec_prefix -- i.e., ignore 'plat_specific'.
     """
     if prefix is None:
-        prefix = plat_specific and EXEC_PREFIX or PREFIX
+        if standard_lib:
+            prefix = plat_specific and BASE_EXEC_PREFIX or BASE_PREFIX
+        else:
+            prefix = plat_specific and EXEC_PREFIX or PREFIX
 
     if os.name == "posix":
         libpython = os.path.join(prefix,
@@ -232,9 +245,9 @@ def get_config_h_filename():
     """Return full pathname of installed pyconfig.h file."""
     if python_build:
         if os.name == "nt":
-            inc_dir = os.path.join(project_base, "PC")
+            inc_dir = os.path.join(_sys_home or project_base, "PC")
         else:
-            inc_dir = project_base
+            inc_dir = _sys_home or project_base
     else:
         inc_dir = get_python_inc(plat_specific=1)
     if get_python_version() < '2.2':
@@ -248,7 +261,8 @@ def get_config_h_filename():
 def get_makefile_filename():
     """Return full pathname of installed Makefile from the Python build."""
     if python_build:
-        return os.path.join(os.path.dirname(sys.executable), "Makefile")
+        return os.path.join(_sys_home or os.path.dirname(sys.executable),
+                                                         "Makefile")
     lib_dir = get_python_lib(plat_specific=0, standard_lib=1)
     config_file = 'config-{}{}'.format(get_python_version(), build_flags)
     return os.path.join(lib_dir, config_file, 'Makefile')

Lib/gettext.py

@@ -55,7 +55,7 @@
            'dgettext', 'dngettext', 'gettext', 'ngettext',
            ]
 
-_default_localedir = os.path.join(sys.prefix, 'share', 'locale')
+_default_localedir = os.path.join(sys.base_prefix, 'share', 'locale')
 
 
 def c2py(plural):

Lib/idlelib/EditorWindow.py

@@ -120,7 +120,7 @@ class EditorWindow(object):
 
     def __init__(self, flist=None, filename=None, key=None, root=None):
         if EditorWindow.help_url is None:
-            dochome =  os.path.join(sys.prefix, 'Doc', 'index.html')
+            dochome =  os.path.join(sys.base_prefix, 'Doc', 'index.html')
             if sys.platform.count('linux'):
                 # look for html docs in a couple of standard places
                 pyver = 'python-docs-' + '%s.%s.%s' % sys.version_info[:3]
@@ -131,13 +131,13 @@ def __init__(self, flist=None, filename=None, key=None, root=None):
                     dochome = os.path.join(basepath, pyver,
                                            'Doc', 'index.html')
             elif sys.platform[:3] == 'win':
-                chmfile = os.path.join(sys.prefix, 'Doc',
+                chmfile = os.path.join(sys.base_prefix, 'Doc',
                                        'Python%s.chm' % _sphinx_version())
                 if os.path.isfile(chmfile):
                     dochome = chmfile
             elif macosxSupport.runningAsOSXApp():
                 # documentation is stored inside the python framework
-                dochome = os.path.join(sys.prefix,
+                dochome = os.path.join(sys.base_prefix,
                         'Resources/English.lproj/Documentation/index.html')
             dochome = os.path.normpath(dochome)
             if os.path.isfile(dochome):

Lib/packaging/command/build_ext.py

@@ -182,7 +182,10 @@ def finalize_options(self):
             # the 'libs' directory is for binary installs - we assume that
             # must be the *native* platform.  But we don't really support
             # cross-compiling via a binary install anyway, so we let it go.
-            self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs'))
+            # Note that we must use sys.base_exec_prefix here rather than
+            # exec_prefix, since the Python libs are not copied to a virtual
+            # environment.
+            self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs'))
             if self.debug:
                 self.build_temp = os.path.join(self.build_temp, "Debug")
             else: