Update docs with 2.12 changes: default Pex lockfile generator, `pants…

….toml` env var interpolation, pip VCS requirements, and Python interpreter constraints (Cherry-pick of #16000) (#16059)

docs/markdown/Python/python/python-interpreter-compatibility.md

@@ -55,28 +55,16 @@ python_sources(
 )
 ```
 
-Pants will merge the constraints from the target's _transitive closure_ when deciding which interpreter to use, meaning that it will look at the constraints of the target, its dependencies, and the dependencies of those dependencies. For example:
-
-- Target A sets `interpreter_constraints==['2.7.*']`.
-- Target B sets `interpreter_contraints=['>=3.5']`, and it depends on Target A.
-- When running `./pants package :b`, Pants will merge the constraints to `['==2.7.*,>=3.5']`. This is impossible to satisfy, so Pants will error.
-
-This means that every dependency of a target must also be compatible with its interpreter constraints. Generally, you will want to be careful that your common `python_source` / `python_sources` targets are compatible with multiple Python versions because they may be depended upon by other targets. Meanwhile, `pex_binary` and `python_test` / `python_tests` targets can have specific constraints because they are (conventionally) never dependencies for other targets. For example:
-
-```python
-python_sources(
-    # Source files are compatible with Python 2.7 or 3.5+.
-    interpreter_constraints=["==2.7.*", ">=3.5"]`,
-)
-
-pex_binary(
-    name="binary",
-    entry_point="app.py",
-    # When merged with the python_sources's constraints, the final result will 
-    # require `>=3.5`.
-    interpreter_constraints=[">=3.5"],
-)
-```
+A particular target's interpreter constraints must be the same or a subset of all of its
+transitive dependencies. For example, a target marked Python 3-only cannot depend on a
+Python 2-only target, which Pants will validate.
+
+It is possible for a target to have more precise interpreter constraints than its transitive
+dependencies. For example, if you have a common helper file `utils.py` that works with both
+Python 2.7 and 3.5 (`['==2.7.*', '>=3.5']`), it is legal for a dependee like `app.py` to use more
+precise constraints like `['==3.7.*']`. It is often useful for helper code to work with multiple
+Python versions, while specific "root" targets (tests, apps, and binaries) have more specific
+constraints.
 
 > 🚧 Pants cannot validate that your interpreter constraints are accurate
 > 

docs/markdown/Python/python/python-third-party-dependencies.md

@@ -228,8 +228,11 @@ Then, use `./pants generate-lockfiles` to generate the lockfile.
 > 
 > As explained at the top of these docs, Pants only uses the subset of the "universe" of your dependencies that is actually needed for a build, such as running tests and packaging a wheel file. This gives fine-grained caching and has other benefits like built packages (e.g. PEX binaries) only including their true dependencies. However, naively, this would mean that you need to resolve dependencies multiple times, which can be slow.
 > 
-> If you use Pex-generated lockfiles (see below), Pants will only install the subset of the lockfile you need for a task. If you use Poetry-generated lockfiles, Pants will first install the entire lockfile and then it will [extract](https://blog.pantsbuild.org/introducing-pants-2-5/) the exact subset needed. 
-> 
+> If you use the default of Pex-generated lockfiles (see below), Pants will only install the
+> subset of the lockfile you need for a task. If you instead use Poetry-generated lockfiles,
+> Pants will first install the entire lockfile and then it
+> will [extract](https://blog.pantsbuild.org/introducing-pants-2-5/) the exact subset needed.
+>
 > This greatly speeds up performance and improves caching for goals like `test`, `run`, `package`, and `repl`.
 
 #### Multiple lockfiles
@@ -347,9 +350,11 @@ To disable lockfiles entirely for a tool, set `[tool].lockfile = "<none>"` for t
 
 ### Pex vs. Poetry for lockfile generation
 
-You should set `[python].lockfile_generator` to either `"pex"` or `"poetry"` in `pants.toml`. The default of `poetry` will change in Pants 2.12.
+Pants defaults to using [Pex](https://pex.readthedocs.io/) to generate lockfiles, but you can
+instead use [Poetry](https://python-poetry.org) by setting `[python].lockfile_generator = "poetry"`
+in `pants.toml`.
 
-We generally recommend using Pex, which has several benefits:
+We generally recommend using the default of Pex, which has several benefits:
 
 1. Supports `[python-repos]` if you have a custom index or repository other than PyPI.
 2. Supports `[GLOBAL].ca_certs_path`.
@@ -469,32 +474,23 @@ mongomock
 
 ### Version control and local requirements
 
-You might be used to using pip's proprietary VCS-style requirements for this, like `git+https://github.com/django/django.git#egg=django`. However, this proprietary format does not work with Pants.
-
-Instead of pip VCS-style requirements:
-
-```
-git+https://github.com/django/django.git#egg=Django
-git+https://github.com/django/django.git@stable/2.1.x#egg=Django
-git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8#egg=Django
-```
-
-Use direct references from [PEP 440](https://www.python.org/dev/peps/pep-0440/#direct-references):
+You can install requirements from version control using two styles:
 
-```
-Django@ git+https://github.com/django/django.git
-Django@ git+https://github.com/django/django.git@stable/2.1.x
-Django@ git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8
-```
+- pip's proprietary VCS-style requirements, e.g.
+  - `git+https://github.com/django/django.git#egg=Django`
+  - `git+https://github.com/django/django.git@stable/2.1.x#egg=Django`
+  - `git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8#egg=Django`
+- direct references from [PEP 440](https://www.python.org/dev/peps/pep-0440/#direct-references), e.g.
+  - `Django@ git+https://github.com/django/django.git`
+  - `Django@ git+https://github.com/django/django.git@stable/2.1.x`
+  - `Django@ git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8`
 
 You can also install from local files using [PEP 440 direct references](https://www.python.org/dev/peps/pep-0440/#direct-references). You must use an absolute path to the file, and you should ensure that the file exists on your machine.
 
 ```
 Django @ file:///Users/pantsbuild/prebuilt_wheels/django-3.1.1-py3-none-any.whl
 ```
 
-Pip still works with these PEP 440-compliant formats, so you won't be losing any functionality by switching to using them.
-
 > 🚧 Local file requirements do not yet work with lockfiles
 > 
 > Pex lockfiles will soon support local file requirements.
@@ -557,20 +553,25 @@ Indexes are assumed to have a nested structure (like <http://pypi.org/simple>),
 
 #### Authenticating to custom repos
 
-To authenticate to these custom repos you may need to provide credentials (such as a username and password) in the URL, that you don't want to expose in your checked-in pants.toml file.  Instead you can do one of the following:
+To authenticate to custom repos, you may need to provide credentials 
+(such as a username and password) in the URL. 
 
-Create a private (not checked-in) [.pants.rc file](doc:options#pantsrc-file) in each user's Pants repo, that sets this config for the user:
+You can use [config file `%(env.ENV_VAR)s` interpolation](doc:options#config-file-interpolation)
+to load the values via environment variables. This avoids checking in sensitive information to
+version control.
 
-```toml .pants.rc
+```toml pants.toml
 [python-repos]
-indexes.add = ["http://$USERNAME:$PASSWORD@my.custom.repo/index"]
+indexes.add = ["http://%(env.INDEX_USERNAME)s:%(INDEX_PASSWORD)s@my.custom.repo/index"]
 ```
 
-Or, set the `indexes` or `repos` config in an environment variable:
+Alternatively, you can hardcode the value in a private (not checked-in)
+[.pants.rc file](doc:options#pantsrc-file) in each user's Pants repo, that sets this config for
+the user:
 
-```shell
-$ export PANTS_PYTHON_REPOS_INDEXES='+["http://$USERNAME:$PASSWORD@my.custom.repo/index"]'
-$ ./pants package ::
+```toml .pants.rc
+[python-repos]
+indexes.add = ["http://$USERNAME:$PASSWORD@my.custom.repo/index"]
 ```
 
 Tip: use `./pants export` to create a virtual environment for IDEs

docs/markdown/Using Pants/concepts/options.md

@@ -66,27 +66,39 @@ Config file entries
 
 Global options are set in the `GLOBAL` section of the config file:
 
-```toml
+```toml pants.toml
 [GLOBAL]
 level = "debug"
 ```
 
 Subsystem options are set in the section named for their scope:
 
-```toml
+```toml pants.toml
 [source]
 root_patterns = ["/src/python"]
 ```
 
 Note that any dashes in the option flag name are converted to underscores: `--multiword-name` becomes `multiword_name`.
 
-Additionally, a few different variables may be interpolated into strings in config files via a `%(var)s` syntax. For example, this expands to the absolute path of a file in the root of your repository:
+### Config file interpolation
 
-```toml
-[GLOBAL]
-pythonpath = ["%(buildroot)s/examples"]
+Environment variables can be interpolated by using the syntax `%(env.ENV_VAR)s`, e.g.:
+
+```toml pants.toml
+[python-repos]
+# This will substitute `%(env.PY_REPO)s` with the value of the environment
+# variable PY_REPO
+indexes.add = ["http://%(env.PY_REPO)s@my.custom.repo/index
 ```
 
+Additionally, a few special values are pre-populated with the `%(var)s` syntax:
+
+- `%(buildroot)s`: absolute path to the root of your repository
+- `%(homedir)s`: equivalent to `$HOME` or `~`
+- `%(user)s`: equivalent to `$USER`
+- `%(pants_distdir)s`: absolute path of the global option `--pants-distdir`, which defaults 
+   to `{buildroot}/dist/`
+
 Option types
 ============
 

docs/markdown/Using Pants/setting-up-an-ide.md

@@ -31,8 +31,8 @@ $ python3 -c "print('PYTHONPATH=\"./' + ':./'.join(\"${ROOTS}\".split()) + ':\$P
 
 See [Use of the PYTHONPATH variable](https://code.visualstudio.com/docs/python/environments#_use-of-the-pythonpath-variable) to learn more about using the `PYTHONPATH` variable in VSCode.
 
-Third-party dependencies (Python)
----------------------------------
+Python third-party dependencies and tools
+-----------------------------------------
 
 To get your editor to understand the repo's third-party dependencies, you will probably want to point it at a virtualenv containing those dependencies.
 
@@ -45,6 +45,19 @@ Wrote virtualenv for the resolve 'python-default' (using CPython==3.9.*) to dist
 
 If you are using the ["resolves" feature for Python lockfiles](doc:python-third-party-dependencies)—which we strongly recommend—Pants will write the virtualenv to `dist/export/python/virtualenvs/<resolve-name>`. If you have multiple resolves, this means that Pants will create one virtualenv per resolve. You can then point your IDE to whichever resolve you want to load at the time.
 
+### Tool virtualenvs
+
+`./pants export` will also create a virtualenv for certain Python tools you use via Pants, like
+formatters like Black and Isort. This allows you to configure your editor to use the same version
+of the tool that Pants uses for workflows like formatting on save.
+
+To disable a certain tool, set its `export` option to `false`, e.g.:
+
+```toml pants.toml
+[black]
+export = false
+```
+
 Generated code
 --------------
 

src/python/pants/notes/2.12.x.md

@@ -162,6 +162,8 @@ The first stable release of the `2.12.x` series, with only documentation changes
 
 * Implement `tailor` goal for Helm charts ([#15192](https://github.com/pantsbuild/pants/pull/15192))
 
+* Write virtualenvs for Python tools as part of `export` ([#15098](https://github.com/pantsbuild/pants/pull/15098))
+
 ### User API Changes
 
 * Upgrade Pex to 2.1.84. ([#15288](https://github.com/pantsbuild/pants/pull/15288))
@@ -194,8 +196,6 @@ The first stable release of the `2.12.x` series, with only documentation changes
 
 * golang: add more cache keys based on tool IDs ([#15205](https://github.com/pantsbuild/pants/pull/15205))
 
-* Mention resolve when partitioning Mypy and Pylint ([#15172](https://github.com/pantsbuild/pants/pull/15172))
-
 ### Performance
 
 * `MyPy` and `Pylint` partition inputs via `CoarsenedTarget` ([#15141](https://github.com/pantsbuild/pants/pull/15141))
@@ -210,6 +210,10 @@ The first stable release of the `2.12.x` series, with only documentation changes
 
 * Show `experimental-bsp` goal in help ([#15208](https://github.com/pantsbuild/pants/pull/15208))
 
+* Mention resolve when partitioning Mypy and Pylint ([#15172](https://github.com/pantsbuild/pants/pull/15172))
+
+* Don't log set up of binary shims ([#15203](https://github.com/pantsbuild/pants/pull/15203))
+
 ## 2.12.0.dev3 (Apr 16, 2022)
 
 ### New Features