[RFC] tox 4

[gaborbernat]

As some of you might probably be aware I've been attempting a virtualenv rewrite (see pypa/virtualenv#1366). To validate my progress on there I've run into the issue of sometimes one needs to run the test suite in a particular Linux distribution. One of the easiest ways of doing this is to run inside a docker image customized for that distro. There's no tox plugin at the moment though that allows one to use docker environments instead of virtual environments. While trying to create this via https://github.com/tox-dev/tox-via-docker I've realized in the internals a lot of things are tied to running things on the local machine via subprocess. Fixing this would mean needing to monkey patch quite a few things within the core. Trying to refactor the internal code to fix this turned into introducing breaking change after breaking change. Therefore, hereby I'm announcing tox 4; that plans to fix this, while also addressing many of the major feature requests we've been putting on the side for a while. Effectively a clean start and complete rewrite.

https://github.com/gaborbernat/tox/tree/docker/src/tox contains a snapshot of this effort. Probably will take all the way to the end of the month for the first version, but opening this issue for a request for comments on the general level.

External facing

0. Python 3.5+ only.
1. Lazy configuration - everything is materialized only when needed (don't ever generate data that will not be used - general speed improvement - notably python discovery previously was generated eagerly at startup independent if was needed or not)
2. built-in wheel build support - no longer generates sdist only
3. library dependency changes are now detected (no longer need to recreate tox-env when adding a new dependency to your library) - use PEP-517 metadata generation to acquire these
4. CLI arguments rewrite - all defaults now are override-able either via global ini then an env var
5. tox now supports sub-commands - still defaults to run sequential the envs (we plan to add additional commands later - e.g. configuration validation):
   • the list envs has migrated to the list sub-command from -a (l shortcut)
   • the show config has migrated to the config sub-command form --showconfig (c shortcut)
   • the run parallel has migrated to run-parallel sub-command form -p (p shortcut)
   • the run sequential has migrated to run sub-command form non other commands (r shortcut)
6. while executing subprocess calls the standard error no longer gets forwarded to the standard output but correctly to the standard error (previously this was only true for
   non captured commands)
7. basepython is now a list, the first successfully detected python will be used to generate python environment
8. Always use isolated build for packaging.

Internal

0. Python 3.5+ only with type annotated code.
1. Separate core configuration concepts from the ini system (to allow introduction of new configuration)
2. so long py my good old friend, use pathlib always
3. Introduce the executor concept - replaces action, generalize to avoid ease of replacement with
4. Generalize tox environment concept to make it python agnostic
5. Separate the packaging environments versus run environments
6. Package environments are tied directly to run environments (multiple run environments may share the same packaging environment)
7. Use the logging framework to log - drop our custom reporter - default log level is INFO
8. Python discovery delegated to virtualenv - due to exposing that in virtualenv is WIP, and dependent on our release we vendor it for now
9. rewrite the internal cache log (log more, structured, phased)

Sample new cache:

{
  "ToxEnv": {
    "name": "py37",
    "type": "VirtualEnvRunner"
  },
  "Python": {
    "version_info": [
      3,
      7,
      4,
      "final",
      0
    ],
    "executable": "/Users/bgabor8/git/github/tox/.tox/dev/bin/python"
  },
  "PythonRun": {
    "deps": [
      "pip==19.2.1"
    ],
    "package_deps": [
      "packaging>=14",
      "pluggy<1,>=0.12.0",
      "appdirs<2,>=1.4.3",
      "virtualenv",
      "importlib-metadata<1,>=0.12; python_version < \"3.8\"",
      "freezegun<1,>=0.3.11",
      "pytest<6,>=4.0.0",
      "pytest-cov<3,>=2.5.1",
      "pytest-mock<2,>=1.10.0"
    ]
  }
}⏎
{
  "ToxEnv": {
    "name": ".package",
    "type": "Pep517VirtualEnvPackageWheel"
  },
  "Python": {
    "version_info": [
      3,
      7,
      4,
      "final",
      0
    ],
    "executable": "/Users/bgabor8/git/github/tox/.tox/dev/bin/python"
  },
  "PythonPackage": {
    "requires": [
      "setuptools >= 40.0.4",
      "setuptools_scm >= 2.0.0, <4",
      "wheel >= 0.29.0"
    ],
    "build-requires": []
  }
}

TODO

• allow overriding all configuration values from the CLI
• index url support for python pip
• introduce the run log concept
• handle provisioning
• make it parallel safe (packaging + logs)
• Make sure we're config compliant with tox 3 (excluding deprecated features) - CLI compliant is best effort
• Allow plugins generating new tox-environments (this will probably require a in-memory config)
• Rewrite documentation (generate configuration from code)

Validate rewrite

• provide a pre-commit env generator plugin

[tox]
requires =
    tox-pre-commit-env == 1.0.0
# pre_commit_version =  >= 1.14.4, < 2

The idea is the above code would be equal with writing the below line and appending fix_lint to the envlist, so we can have shorter configurations:

[testenv:fix_lint]
description = format the code base to adhere to our styles, and complain about what we cannot do automatically
basepython = python3.7
passenv = {[testenv]passenv}
          # without PROGRAMDATA cloning using git for Windows will fail with an
          # `error setting certificate verify locations` error
          PROGRAMDATA
extras = lint
deps = pre-commit >= 1.14.4, < 2
skip_install = True
commands = pre-commit run --all-files --show-diff-on-failure
           python -c 'import pathlib; print("hint: run \{\} install to add checks as pre-commit hook".format(pathlib.Path(r"{envdir}") / "bin" / "pre-commit"))'

• provide a sphinx doc env generator plugin
• Provide a tox environment that uses Docker images instead of virtual environments (this will validate the internal refactor)
• migrate some popular tox plugins to the new system (tox-travis + tox-pipenv + tox-conda + tox-pyenv + tox-current-env)

@asottile @obestwalter let me know your concerns/requests, thanks! I'm also inviting the communities thoughts, cc @ssbarnea.

[jtrakk]

I'd really like to see Poetry support. Currently the way I use Poetry with tox is via this brilliant hack by @canassa of using a fake deps:

install_command = poetry {packages}
deps = install

Outside tox, the way Poetry is used is poetry install, which creates a virtualenv and installs the package and its development dependencies, and then poetry run pytest.

[asottile]

@jtrakk as long as I'm reading this correctly, this will better enable a plugin to add poetry support to tox (it's ~kinda difficult right now, though probably doable with the venv hooks)

---

as for the proposal here are my thoughts

• py35+ seems fine, though we're going to potentially break a lot of old CI. py36+ to me seems more enticing but if we're going on pytest's precedent then 3.5 is probably a good lower bound
• happy we're killing py lib, I was slowly chipping away at it but I a rewrite'll purge it real good
• curious how the dependency changes are going to be detected (and if it'll be fast!), I guess I'll have to wait and see
• "Use the logging framework to log" - personally I find logging way too easy to end up with sneaky globals, but it'll probably be fine
• the pathlib bits in the pre-commit example maybe don't need to happen any more as pre-commit added some messaging directly :)

[gaborbernat]

For now configurations are calculated at first access and cached indefinitely afterwards. We probably can add some kind of auto update if dependent configurations change later on. I've been working the last few weeks on adding tests. I got to around 70 percent now, takes some time though to make it feature full and deploy ready. It can run itself for now (not yet in parallel). A side effect of the rewrite is also a rewrite of the test suite, I'm trying to make it really fast (sub minute test suite is my hope/goal) for unit tests. Maybe two more for unit Plus integration tests.

[blueyed]

Just 2 cents: targetting py36+ is better for type annotations (very useful in a big rewrite I assume), and it will likely take some time anyway (py35 is only supported until 2020-09-13 officially). It could still be used to run py35 envs (I assume).

[gaborbernat]

You can use type comments, took me like 1 hour to make tests pass on 3.5 once they were on 3.8. so using 3.5+ I found to be enough for now.

[obestwalter]

Sorry, this completely slipped by. Looks like a great initiative 👍

I will take some time as soon as I can to have a closer look and give some detailed feedback.

[gaborbernat]

@obestwalter that's alright, I have a POC version running at #1400, but for now got put on the backlog, will continue after pypa/virtualenv#1377 (probably in December).

[Zac-HD]

3. library dependency changes are now detected (no longer need to recreate tox-env when adding a new dependency to your library) - use PEP-517 metadata generation to acquire these

That's #149, right? Very exciting!

[gaborbernat]

Partially, for now works for dependencies specified by build-backends; aka libraries; but plan is to support requirement/pipfiles too 👍

[gaborbernat]

For who is following this I've picked this up again via the rewrite branch 👍 now that virtualenv rewrite is done.