Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add topic guide: VCS Support #10099

Merged
merged 4 commits into from
Jul 12, 2021
Merged

Add topic guide: VCS Support #10099

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ad8a1b8
Add topic guide: VCS Support
pradyunsg Jun 25, 2021
9cd3d5d
Drop old VCS support section and update links
pradyunsg Jun 25, 2021
80285c6
Correctly look up `os.devnull` reference
pradyunsg Jun 25, 2021
c1295af
Address review comments
pradyunsg Jul 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 4 additions & 165 deletions docs/html/cli/pip_install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ for the name and project version (this is in theory slightly less reliable
than using the ``egg_info`` command, but avoids downloading and processing
unnecessary numbers of files).

Any URL may use the ``#egg=name`` syntax (see :ref:`VCS Support`) to
Any URL may use the ``#egg=name`` syntax (see :doc:`../topics/vcs-support`) to
explicitly state the project name.

Satisfying Requirements
Expand Down Expand Up @@ -372,168 +372,7 @@ that enables installation of pre-releases and development releases.
VCS Support
-----------

pip supports installing from Git, Mercurial, Subversion and Bazaar, and detects
the type of VCS using URL prefixes: ``git+``, ``hg+``, ``svn+``, and ``bzr+``.

pip requires a working VCS command on your path: ``git``, ``hg``, ``svn``, or
``bzr``.

VCS projects can be installed in :ref:`editable mode <editable-installs>` (using
the :ref:`--editable <install_--editable>` option) or not.

* For editable installs, the clone location by default is ``<venv
path>/src/SomeProject`` in virtual environments, and
``<cwd>/src/SomeProject``
for global installs. The :ref:`--src <install_--src>` option can be used to
modify this location.
* For non-editable installs, the project is built locally in a temp dir and then
installed normally. Note that if a satisfactory version of the package is
already installed, the VCS source will not overwrite it without an
``--upgrade`` flag. VCS requirements pin the package version (specified
in the ``setup.py`` file) of the target commit, not necessarily the commit
itself.
* The :ref:`pip freeze` subcommand will record the VCS requirement specifier
(referencing a specific commit) if and only if the install is done using the
editable option.

The "project name" component of the URL suffix ``egg=<project name>``
is used by pip in its dependency logic to identify the project prior
to pip downloading and analyzing the metadata. For projects
where ``setup.py`` is not in the root of project, the "subdirectory" component
is used. The value of the "subdirectory" component should be a path starting
from the root of the project to where ``setup.py`` is located.

If your repository layout is::

pkg_dir
├── setup.py # setup.py for package "pkg"
└── some_module.py
other_dir
└── some_file
some_other_file

Then, to install from this repository, the syntax would be:

.. tab:: Unix/macOS

.. code-block:: shell

python -m pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"

.. tab:: Windows

.. code-block:: shell

py -m pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"


Git
^^^

pip currently supports cloning over ``git``, ``git+http``, ``git+https``,
``git+ssh``, ``git+git`` and ``git+file``.

.. warning::

Note that the use of ``git``, ``git+git``, and ``git+http`` is discouraged.
The former two use `the Git Protocol`_, which lacks authentication, and HTTP is
insecure due to lack of TLS based encryption.

Here are the supported forms::

[-e] git+http://git.example.com/MyProject#egg=MyProject
[-e] git+https://git.example.com/MyProject#egg=MyProject
[-e] git+ssh://git.example.com/MyProject#egg=MyProject
[-e] git+file:///home/user/projects/MyProject#egg=MyProject

Passing a branch name, a commit hash, a tag name or a git ref is possible like so::

[-e] git+https://git.example.com/MyProject.git@main#egg=MyProject
[-e] git+https://git.example.com/MyProject.git@v1.0#egg=MyProject
[-e] git+https://git.example.com/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject
[-e] git+https://git.example.com/MyProject.git@refs/pull/123/head#egg=MyProject

When passing a commit hash, specifying a full hash is preferable to a partial
hash because a full hash allows pip to operate more efficiently (e.g. by
making fewer network calls).

.. _`the Git Protocol`: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

Mercurial
^^^^^^^^^

The supported schemes are: ``hg+file``, ``hg+http``, ``hg+https``,
``hg+static-http``, and ``hg+ssh``.

Here are the supported forms::

[-e] hg+http://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+https://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+ssh://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+file:///home/user/projects/MyProject#egg=MyProject

You can also specify a revision number, a revision hash, a tag name or a local
branch name like so::

[-e] hg+http://hg.example.com/MyProject@da39a3ee5e6b#egg=MyProject
[-e] hg+http://hg.example.com/MyProject@2019#egg=MyProject
[-e] hg+http://hg.example.com/MyProject@v1.0#egg=MyProject
[-e] hg+http://hg.example.com/MyProject@special_feature#egg=MyProject

Subversion
^^^^^^^^^^

pip supports the URL schemes ``svn``, ``svn+svn``, ``svn+http``, ``svn+https``, ``svn+ssh``.

Here are some of the supported forms::

[-e] svn+https://svn.example.com/MyProject#egg=MyProject
[-e] svn+ssh://svn.example.com/MyProject#egg=MyProject
[-e] svn+ssh://user@svn.example.com/MyProject#egg=MyProject

You can also give specific revisions to an SVN URL, like so::

[-e] svn+svn://svn.example.com/svn/MyProject#egg=MyProject
[-e] svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject

which will check out revision 2019. ``@{20080101}`` would also check
out the revision from 2008-01-01. You can only check out specific
revisions using ``-e svn+...``.

Bazaar
^^^^^^

pip supports Bazaar using the ``bzr+http``, ``bzr+https``, ``bzr+ssh``,
``bzr+sftp``, ``bzr+ftp`` and ``bzr+lp`` schemes.

Here are the supported forms::

[-e] bzr+http://bzr.example.com/MyProject/trunk#egg=MyProject
[-e] bzr+sftp://user@example.com/MyProject/trunk#egg=MyProject
[-e] bzr+ssh://user@example.com/MyProject/trunk#egg=MyProject
[-e] bzr+ftp://user@example.com/MyProject/trunk#egg=MyProject
[-e] bzr+lp:MyProject#egg=MyProject

Tags or revisions can be installed like so::

[-e] bzr+https://bzr.example.com/MyProject/trunk@2019#egg=MyProject
[-e] bzr+http://bzr.example.com/MyProject/trunk@v1.0#egg=MyProject

Using Environment Variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since version 10, pip also makes it possible to use environment variables which
makes it possible to reference private repositories without having to store
access tokens in the requirements file. For example, a private git repository
allowing Basic Auth for authentication can be refenced like this::

[-e] git+http://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject#egg=MyProject
[-e] git+https://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject#egg=MyProject

.. note::

Only ``${VARIABLE}`` is supported, other formats like ``$VARIABLE`` or
``%VARIABLE%`` won't work.
This is now covered in :doc:`../topics/vcs-support`.

Finding Packages
----------------
Expand Down Expand Up @@ -801,7 +640,7 @@ You can install local projects or VCS projects in "editable" mode:
py -m pip install -e git+http://repo/my_project.git#egg=SomeProject


(See the :ref:`VCS Support` section above for more information on VCS-related syntax.)
(See the :doc:`../topics/vcs-support` section above for more information on VCS-related syntax.)

For local projects, the "SomeProject.egg-info" directory is created relative to
the project path. This is one advantage over just using ``setup.py develop``,
Expand Down Expand Up @@ -992,7 +831,7 @@ Examples
py -m pip install SomeProject@git+https://git.repo/some_pkg.git@1.3.1


#. Install a project from VCS in "editable" mode. See the sections on :ref:`VCS Support <VCS Support>` and :ref:`Editable Installs <editable-installs>`.
#. Install a project from VCS in "editable" mode. See the sections on :doc:`../topics/vcs-support` and :ref:`Editable Installs <editable-installs>`.

.. tab:: Unix/macOS

Expand Down
2 changes: 1 addition & 1 deletion docs/html/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ $ pip install git+https://github.com/pypa/sampleproject.git@main
Successfully installed sampleproject
```

See {ref}`VCS Support` for more information about this syntax.
See {doc}`topics/vcs-support` for more information about this syntax.

### Install a package from a distribution file

Expand Down
4 changes: 2 additions & 2 deletions docs/html/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ Usually, pip is automatically installed if you are:
If your Python environment does not have pip installed, there are 2 mechanisms
to install pip supported directly by pip's maintainers:

- [`ensurepip`](#using-ensurepip)
- [`get-pip.py`](#using-get-pip-py)
- [`ensurepip`](#ensurepip)
- [`get-pip.py`](#get-pip-py)

### `ensurepip`

Expand Down
2 changes: 1 addition & 1 deletion docs/html/topics/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Site

Additionally, the environment variable `PIP_CONFIG_FILE` can be used to specify
a configuration file that's loaded first, and whose values are overridden by
the values set in the aforementioned files. Setting this to {ref}`os.devnull`
the values set in the aforementioned files. Setting this to {any}`os.devnull`
pradyunsg marked this conversation as resolved.
Show resolved Hide resolved
disables the loading of _all_ configuration files.

### Loading order
Expand Down
1 change: 1 addition & 0 deletions docs/html/topics/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ This section of the documentation is currently being fleshed out. See
authentication
caching
configuration
vcs-support
```
162 changes: 162 additions & 0 deletions docs/html/topics/vcs-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# VCS Support

pip supports installing from various version control systems (VCS).
This support requires a working executable to be available (for the version
control system being used). It is used through URL prefixes:

- Git -- `git+`
- Mercurial -- `hg+`
- Subversion -- `svn+`
- Bazaar -- `bzr+`

## Supported VCS

### Git

The supported schemes are `git+file`, `git+https`, `git+ssh`, `git+http`,
`git+git` and `git`. Here are some of the supported forms:

```none
git+ssh://git.example.com/MyProject#egg=MyProject
git+file:///home/user/projects/MyProject#egg=MyProject
git+https://git.example.com/MyProject#egg=MyProject
```

```{warning}
The use of `git`, `git+git`, and `git+http` schemes is discouraged.
The former two use [the Git Protocol], which lacks authentication, and HTTP is
insecure due to lack of TLS based encryption.
```
Comment on lines +25 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should just mention git+file, git+https, and git+ssh in the opening paragraph, and add a “oh btw we also recognise these other three but they are bad anyway” note here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe. Let's defer this for a follow up PR though, since one of the things I like about the way it's currently written is that all the sections have the exact same start that lists all the things.


[the Git Protocol]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

It is also possible to specify a "git ref" such as branch name, a commit hash or
a tag name:

```none
git+https://git.example.com/MyProject.git@master#egg=MyProject
git+https://git.example.com/MyProject.git@v1.0#egg=MyProject
git+https://git.example.com/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject
git+https://git.example.com/MyProject.git@refs/pull/123/head#egg=MyProject
```

When passing a commit hash, specifying a full hash is preferable to a partial
hash because a full hash allows pip to operate more efficiently (e.g. by
making fewer network calls).

### Mercurial

The supported schemes are `hg+file`, `hg+http`, `hg+https`, `hg+ssh`
and `hg+static-http`. Here are some of the supported forms:

```
hg+http://hg.myproject.org/MyProject#egg=MyProject
hg+https://hg.myproject.org/MyProject#egg=MyProject
hg+ssh://hg.myproject.org/MyProject#egg=MyProject
hg+file:///home/user/projects/MyProject#egg=MyProject
```

It is also possible to specify a revision number, a revision hash, a tag name
or a local branch name:

```none
hg+http://hg.example.com/MyProject@da39a3ee5e6b#egg=MyProject
hg+http://hg.example.com/MyProject@2019#egg=MyProject
hg+http://hg.example.com/MyProject@v1.0#egg=MyProject
hg+http://hg.example.com/MyProject@special_feature#egg=MyProject
```

### Subversion

The supported schemes are `svn`, `svn+svn`, `svn+http`, `svn+https` and
`svn+ssh`. Here are some of the supported forms:

```none
svn+https://svn.example.com/MyProject#egg=MyProject
svn+ssh://svn.example.com/MyProject#egg=MyProject
svn+ssh://user@svn.example.com/MyProject#egg=MyProject
```

You can also give specific revisions to an SVN URL, like so:

```none
-e svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject
-e svn+http://svn.example.com/svn/MyProject/trunk@{20080101}#egg=MyProject
```

Note that you need to use [Editable VCS installs](#editable-vcs-installs) for
using specific revisions from Subversion.

### Bazaar

The supported schemes are `bzr+http`, `bzr+https`, `bzr+ssh`, `bzr+sftp`,
`bzr+ftp` and `bzr+lp`. Here are the supported forms:

```none
bzr+http://bzr.example.com/MyProject/trunk#egg=MyProject
bzr+sftp://user@example.com/MyProject/trunk#egg=MyProject
bzr+ssh://user@example.com/MyProject/trunk#egg=MyProject
bzr+ftp://user@example.com/MyProject/trunk#egg=MyProject
bzr+lp:MyProject#egg=MyProject
```

Tags or revisions can be installed like so:

```none
bzr+https://bzr.example.com/MyProject/trunk@2019#egg=MyProject
bzr+http://bzr.example.com/MyProject/trunk@v1.0#egg=MyProject
```

(editable-vcs-installs)=

## Editable VCS installs

VCS projects can be installed in {ref}`editable mode <editable-installs>` (using
the {ref}`--editable <install_--editable>` option) or not.

- The default clone location (for editable installs) is:

- `<venv path>/src/SomeProject` in virtual environments
- `<cwd>/src/SomeProject` for global Python installs

The {ref}`--src <install_--src>` option can be used to modify this location.

- For non-editable installs, the project is built locally in a temp dir and then
installed normally.

Note that if a satisfactory version of the package is already installed, the
VCS source will not overwrite it without an `--upgrade` flag. Further, pip
looks at the package version, at the target revision to determine what action to
take on the VCS requirement (not the commit itself).

The {ref}`pip freeze` subcommand will record the VCS requirement specifier
(referencing a specific commit) only if the install is done with the editable
option.

## URL fragments

pip looks at 2 fragments for VCS URLs:

- `egg`: For specifying the "project name" for use in pip's dependency
resolution logic. eg: `egg=project_name`
- `subdirectory`: For specifying the path to the Python package, when it is not
in the root of the VCS directory. eg: `pkg_dir`

````{admonition} Example
If your repository layout is:

```
pkg_dir
├── setup.py # setup.py for package "pkg"
└── some_module.py
other_dir
└── some_file
some_other_file
```

Then, to install from this repository, the syntax would be:

```{pip-cli}
$ pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
```
````
Loading