Merge pull request #5743 from pfmoore/pep517

PEP 517 implementation

docs/html/reference/pip.rst

@@ -71,13 +71,13 @@ when decision is needed.
 Build System Interface
 ======================
 
-Pip builds packages by invoking the build system. Presently, the only supported
-build system is ``setuptools``, but in the future, pip will support :pep:`517`
-which allows projects to specify an alternative build system in a
-``pyproject.toml`` file.  As well as package building, the build system is also
-invoked to install packages direct from source.  This is handled by invoking
-the build system to build a wheel, and then installing from that wheel.  The
-built wheel is cached locally by pip to avoid repeated identical builds.
+Pip builds packages by invoking the build system. By default, builds will use
+``setuptools``, but if a project specifies a different build system using a
+``pyproject.toml`` file, as per :pep:`517`, pip will use that instead.  As well
+as package building, the build system is also invoked to install packages
+direct from source.  This is handled by invoking the build system to build a
+wheel, and then installing from that wheel.  The built wheel is cached locally
+by pip to avoid repeated identical builds.
 
 The current interface to the build system is via the ``setup.py`` command line
 script - all build actions are defined in terms of the specific ``setup.py``
@@ -86,13 +86,16 @@ command line that will be run to invoke the required action.
 Setuptools Injection
 ~~~~~~~~~~~~~~~~~~~~
 
-As noted above, the supported build system is ``setuptools``. However, not all
-packages use ``setuptools`` in their build scripts. To support projects that
-use "pure ``distutils``", pip injects ``setuptools`` into ``sys.modules``
-before invoking ``setup.py``. The injection should be transparent to
-``distutils``-based projects, but 3rd party build tools wishing to provide a
-``setup.py`` emulating the commands pip requires may need to be aware that it
-takes place.
+When :pep:`517` is not used, the supported build system is ``setuptools``.
+However, not all packages use ``setuptools`` in their build scripts. To support
+projects that use "pure ``distutils``", pip injects ``setuptools`` into
+``sys.modules`` before invoking ``setup.py``. The injection should be
+transparent to ``distutils``-based projects, but 3rd party build tools wishing
+to provide a ``setup.py`` emulating the commands pip requires may need to be
+aware that it takes place.
+
+Projects using :pep:`517` *must* explicitly use setuptools - pip does not do
+the above injection process in this case.
 
 Build System Output
 ~~~~~~~~~~~~~~~~~~~
@@ -113,13 +116,20 @@ unexpected byte sequences to Python-style hexadecimal escape sequences
 (``"\x80\xff"``, etc). However, it is still possible for output to be displayed
 using an incorrect encoding (mojibake).
 
-PEP 518 Support
-~~~~~~~~~~~~~~~
+Under :pep:`517`, handling of build tool output is the backend's responsibility,
+and pip simply displays the output produced by the backend. (Backends, however,
+will likely still have to address the issues described above).
+
+PEP 517 and 518 Support
+~~~~~~~~~~~~~~~~~~~~~~~
 
-As of 10.0, pip supports projects declaring dependencies that are required at
-install time using a ``pyproject.toml`` file, in the form described in
-:pep:`518`. When building a project, pip will install the required dependencies
-locally, and make them available to the build process.
+As of version 10.0, pip supports projects declaring dependencies that are
+required at install time using a ``pyproject.toml`` file, in the form described
+in :pep:`518`. When building a project, pip will install the required
+dependencies locally, and make them available to the build process.
+Furthermore, from version 19.0 onwards, pip supports projects specifying the
+build backend they use in ``pyproject.toml``, in the form described in
+:pep:`517`.
 
 When making build requirements available, pip does so in an *isolated
 environment*. That is, pip does not install those requirements into the user's
@@ -137,24 +147,49 @@ can be problematic. If this is the case, pip provides a
 flag are responsible for ensuring the build environment is managed
 appropriately.
 
-.. _pep-518-limitations:
-
-**Limitations**:
+By default, pip will continue to use the legacy (``setuptools`` based) build
+processing for projects that do not have a ``pyproject.toml`` file. Projects
+with a ``pyproject.toml`` file will use a :pep:`517` backend. Projects with
+a ``pyproject.toml`` file, but which don't have a ``build-system`` section,
+will be assumed to have the following backend settings::
+
+    [build-system]
+    requires = ["setuptools>=40.2.0", "wheel"]
+    build-backend = "setuptools.build_meta"
+
+.. note::
+
+    ``setuptools`` 40.2.0 is the first version of setuptools with full
+    :pep:`517` support.
+
+If a project has ``[build-system]``, but no ``build-backend``, pip will use
+``setuptools.build_meta``, but will assume the project requirements include
+``setuptools>=40.2.0`` and ``wheel`` (and will report an error if not).
+
+If a user wants to explicitly request :pep:`517` handling even though a project
+doesn't have a ``pyproject.toml`` file, this can be done using the
+``--use-pep517`` command line option. Similarly, to request legacy processing
+even though ``pyproject.toml`` is present, the ``--no-use-pep517`` option is
+available (although obviously it is an error to choose ``--no-use-pep517`` if
+the project has no ``setup.py``, or explicitly requests a build backend). As
+with other command line flags, pip recognises the ``PIP_USE_PEP517``
+environment veriable and a ``use-pep517`` config file option (set to true or
+false) to set this option globally. Note that overriding pip's choice of
+whether to use :pep:`517` processing in this way does *not* affect whether pip
+will use an isolated build environment (which is controlled via
+``--no-build-isolation`` as noted above).
+
+Except in the case noted above (projects with no :pep:`518` ``[build-system]``
+section in ``pyproject.toml``), pip will never implicitly install a build
+system. Projects **must** ensure that the correct build system is listed in
+their ``requires`` list (this applies even if pip assumes that the
+``setuptools`` backend is being used, as noted above).
 
-* until :pep:`517` support is added, ``setuptools`` and ``wheel`` **must** be
-  included in the list of build requirements: pip will assume these as default,
-  but will not automatically add them to the list of build requirements if
-  explicitly defined in ``pyproject.toml``.
+.. _pep-518-limitations:
 
-* the current implementation only support installing build requirements from
-  wheels: this is a technical limitation of the implementation - source
-  installs would require a build step of their own, potentially recursively
-  triggering another :pep:`518` dependency installation process. The possible
-  unbounded recursion involved was not considered acceptable, and so
-  installation of build dependencies from source has been disabled until a safe
-  resolution of this issue is found.
+**Historical Limitations**:
 
-* ``pip<18.0``: only support installing build requirements from wheels, and
+* ``pip<18.0``: only supports installing build requirements from wheels, and
   does not support the use of environment markers and extras (only version
   specifiers are respected).
 

news/5743.feature

@@ -0,0 +1 @@
+Implement PEP 517 (allow projects to specify a build backend via pyproject.toml).

src/pip/_internal/build_env.py

@@ -5,6 +5,7 @@
 import os
 import sys
 import textwrap
+from collections import OrderedDict
 from distutils.sysconfig import get_python_lib
 from sysconfig import get_paths
 
@@ -18,6 +19,25 @@
 logger = logging.getLogger(__name__)
 
 
+class _Prefix:
+
+    def __init__(self, path):
+        self.path = path
+        self.setup = False
+        self.bin_dir = get_paths(
+            'nt' if os.name == 'nt' else 'posix_prefix',
+            vars={'base': path, 'platbase': path}
+        )['scripts']
+        # Note: prefer distutils' sysconfig to get the
+        # library paths so PyPy is correctly supported.
+        purelib = get_python_lib(plat_specific=0, prefix=path)
+        platlib = get_python_lib(plat_specific=1, prefix=path)
+        if purelib == platlib:
+            self.lib_dirs = [purelib]
+        else:
+            self.lib_dirs = [purelib, platlib]
+
+
 class BuildEnvironment(object):
     """Creates and manages an isolated environment to install build deps
     """
@@ -26,86 +46,113 @@ def __init__(self):
         self._temp_dir = TempDirectory(kind="build-env")
         self._temp_dir.create()
 
-    @property
-    def path(self):
-        return self._temp_dir.path
-
-    def __enter__(self):
-        self.save_path = os.environ.get('PATH', None)
-        self.save_pythonpath = os.environ.get('PYTHONPATH', None)
-        self.save_nousersite = os.environ.get('PYTHONNOUSERSITE', None)
-
-        install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
-        install_dirs = get_paths(install_scheme, vars={
-            'base': self.path,
-            'platbase': self.path,
-        })
-
-        scripts = install_dirs['scripts']
-        if self.save_path:
-            os.environ['PATH'] = scripts + os.pathsep + self.save_path
-        else:
-            os.environ['PATH'] = scripts + os.pathsep + os.defpath
-
-        # Note: prefer distutils' sysconfig to get the
-        # library paths so PyPy is correctly supported.
-        purelib = get_python_lib(plat_specific=0, prefix=self.path)
-        platlib = get_python_lib(plat_specific=1, prefix=self.path)
-        if purelib == platlib:
-            lib_dirs = purelib
-        else:
-            lib_dirs = purelib + os.pathsep + platlib
-        if self.save_pythonpath:
-            os.environ['PYTHONPATH'] = lib_dirs + os.pathsep + \
-                self.save_pythonpath
-        else:
-            os.environ['PYTHONPATH'] = lib_dirs
-
-        os.environ['PYTHONNOUSERSITE'] = '1'
-
-        # Ensure .pth files are honored.
-        with open(os.path.join(purelib, 'sitecustomize.py'), 'w') as fp:
+        self._prefixes = OrderedDict((
+            (name, _Prefix(os.path.join(self._temp_dir.path, name)))
+            for name in ('normal', 'overlay')
+        ))
+
+        self._bin_dirs = []
+        self._lib_dirs = []
+        for prefix in reversed(list(self._prefixes.values())):
+            self._bin_dirs.append(prefix.bin_dir)
+            self._lib_dirs.extend(prefix.lib_dirs)
+
+        # Customize site to:
+        # - ensure .pth files are honored
+        # - prevent access to system site packages
+        system_sites = {
+            os.path.normcase(site) for site in (
+                get_python_lib(plat_specific=0),
+                get_python_lib(plat_specific=1),
+            )
+        }
+        self._site_dir = os.path.join(self._temp_dir.path, 'site')
+        if not os.path.exists(self._site_dir):
+            os.mkdir(self._site_dir)
+        with open(os.path.join(self._site_dir, 'sitecustomize.py'), 'w') as fp:
             fp.write(textwrap.dedent(
                 '''
-                import site
-                site.addsitedir({!r})
+                import os, site, sys
+
+                # First, drop system-sites related paths.
+                original_sys_path = sys.path[:]
+                known_paths = set()
+                for path in {system_sites!r}:
+                    site.addsitedir(path, known_paths=known_paths)
+                system_paths = set(
+                    os.path.normcase(path)
+                    for path in sys.path[len(original_sys_path):]
+                )
+                original_sys_path = [
+                    path for path in original_sys_path
+                    if os.path.normcase(path) not in system_paths
+                ]
+                sys.path = original_sys_path
+
+                # Second, add lib directories.
+                # ensuring .pth file are processed.
+                for path in {lib_dirs!r}:
+                    assert not path in sys.path
+                    site.addsitedir(path)
                 '''
-            ).format(purelib))
+            ).format(system_sites=system_sites, lib_dirs=self._lib_dirs))
 
-        return self.path
+    def __enter__(self):
+        self._save_env = {
+            name: os.environ.get(name, None)
+            for name in ('PATH', 'PYTHONNOUSERSITE', 'PYTHONPATH')
+        }
+
+        path = self._bin_dirs[:]
+        old_path = self._save_env['PATH']
+        if old_path:
+            path.extend(old_path.split(os.pathsep))
+
+        pythonpath = [self._site_dir]
+
+        os.environ.update({
+            'PATH': os.pathsep.join(path),
+            'PYTHONNOUSERSITE': '1',
+            'PYTHONPATH': os.pathsep.join(pythonpath),
+        })
 
     def __exit__(self, exc_type, exc_val, exc_tb):
-        def restore_var(varname, old_value):
+        for varname, old_value in self._save_env.items():
             if old_value is None:
                 os.environ.pop(varname, None)
             else:
                 os.environ[varname] = old_value
 
-        restore_var('PATH', self.save_path)
-        restore_var('PYTHONPATH', self.save_pythonpath)
-        restore_var('PYTHONNOUSERSITE', self.save_nousersite)
-
     def cleanup(self):
         self._temp_dir.cleanup()
 
-    def missing_requirements(self, reqs):
-        """Return a list of the requirements from reqs that are not present
+    def check_requirements(self, reqs):
+        """Return 2 sets:
+            - conflicting requirements: set of (installed, wanted) reqs tuples
+            - missing requirements: set of reqs
         """
-        missing = []
-        with self:
-            ws = WorkingSet(os.environ["PYTHONPATH"].split(os.pathsep))
+        missing = set()
+        conflicting = set()
+        if reqs:
+            ws = WorkingSet(self._lib_dirs)
             for req in reqs:
                 try:
                     if ws.find(Requirement.parse(req)) is None:
-                        missing.append(req)
-                except VersionConflict:
-                    missing.append(req)
-            return missing
-
-    def install_requirements(self, finder, requirements, message):
+                        missing.add(req)
+                except VersionConflict as e:
+                    conflicting.add((str(e.args[0].as_requirement()),
+                                     str(e.args[1])))
+        return conflicting, missing
+
+    def install_requirements(self, finder, requirements, prefix, message):
+        prefix = self._prefixes[prefix]
+        assert not prefix.setup
+        prefix.setup = True
+        if not requirements:
+            return
         args = [
             sys.executable, os.path.dirname(pip_location), 'install',
-            '--ignore-installed', '--no-user', '--prefix', self.path,
+            '--ignore-installed', '--no-user', '--prefix', prefix.path,
             '--no-warn-script-location',
         ]
         if logger.getEffectiveLevel() <= logging.DEBUG:
@@ -150,5 +197,5 @@ def __exit__(self, exc_type, exc_val, exc_tb):
     def cleanup(self):
         pass
 
-    def install_requirements(self, finder, requirements, message):
+    def install_requirements(self, finder, requirements, prefix, message):
         raise NotImplementedError()

src/pip/_internal/cli/base_command.py

@@ -232,6 +232,7 @@ def populate_requirement_set(requirement_set,  # type: RequirementSet
         for req in args:
             req_to_add = install_req_from_line(
                 req, None, isolated=options.isolated_mode,
+                use_pep517=options.use_pep517,
                 wheel_cache=wheel_cache
             )
             req_to_add.is_direct = True
@@ -241,6 +242,7 @@ def populate_requirement_set(requirement_set,  # type: RequirementSet
             req_to_add = install_req_from_editable(
                 req,
                 isolated=options.isolated_mode,
+                use_pep517=options.use_pep517,
                 wheel_cache=wheel_cache
             )
             req_to_add.is_direct = True
@@ -250,7 +252,8 @@ def populate_requirement_set(requirement_set,  # type: RequirementSet
             for req_to_add in parse_requirements(
                     filename,
                     finder=finder, options=options, session=session,
-                    wheel_cache=wheel_cache):
+                    wheel_cache=wheel_cache,
+                    use_pep517=options.use_pep517):
                 req_to_add.is_direct = True
                 requirement_set.add_requirement(req_to_add)
         # If --require-hashes was a line in a requirements file, tell