Allow disabling plugins via CLI

[ssbarnea]

Issue

# inside tox own codebase:
tox -e 3.13 -- -k test_provision_requires_ok

Note that version of python is irrelevant, as it does reproduce with all supported versions.

I also tried to modify the failing test to compensate for the missing pip, but it still failed with another error, now complaining about missing hatchling... and adding hatchling did not address this one. So we might have more than one bug here.

proj = tox_project({"tox.ini": "[tox]\nrequires=demo-pkg-inline\n[testenv]\npackage=skip\ndeps=pip"})

Environment

Provide at least:

• OS:

Output of pip list of the host Python, where tox is installed

Output of running tox

Output of tox -rvv

❯ tox -e 3.10 -- -k test_provision_requires_ok
3.10: venv> /Users/ssbarnea/.config/mise/installs/python/3.13.0/bin/uv venv -p 3.10 --allow-existing /Users/ssbarnea/code/os/tox/.tox/3.10
3.10: install_dependency-groups> /Users/ssbarnea/.config/mise/installs/python/3.13.0/bin/uv pip install 'build[virtualenv]>=1.2.2.post1' 'covdefaults>=2.3' 'detect-test-pollution>=1.2' 'devpi-process>=1.0.2' 'diff-cover>=9.2' 'distlib>=0.3.9' 'flaky>=3.8.1' 'hatch-vcs>=0.4' 'hatchling>=1.26.3' 'psutil>=6.1' 'pytest-cov>=5' 'pytest-mock>=3.14' 'pytest-xdist>=3.6.1' 'pytest>=8.3.3' 're-assert>=1.1' 'setuptools>=75.1; python_version <= "3.8"' 'setuptools>=75.6; python_version > "3.8"' 'time-machine>=2.15; implementation_name != "pypy"' 'wheel>=0.45'
.pkg: _optional_hooks> python /Users/ssbarnea/.config/mise/installs/python/3.13.0/lib/python3.13/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_wheel> python /Users/ssbarnea/.config/mise/installs/python/3.13.0/lib/python3.13/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_editable> python /Users/ssbarnea/.config/mise/installs/python/3.13.0/lib/python3.13/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: build_wheel> python /Users/ssbarnea/.config/mise/installs/python/3.13.0/lib/python3.13/site-packages/pyproject_api/_backend.py True hatchling.build
3.10: install_package_deps> /Users/ssbarnea/.config/mise/installs/python/3.13.0/bin/uv pip install 'cachetools>=5.5' 'chardet>=5.2' 'colorama>=0.4.6' 'filelock>=3.16.1' 'packaging>=24.2' 'platformdirs>=4.3.6' 'pluggy>=1.5' 'pyproject-api>=1.8' 'tomli>=2.1; python_version < "3.11"' 'typing-extensions>=4.12.2; python_version < "3.11"' 'virtualenv>=20.27.1'
3.10: install_package> /Users/ssbarnea/.config/mise/installs/python/3.13.0/bin/uv pip install --reinstall --no-deps tox@/Users/ssbarnea/code/os/tox/.tox/.tmp/package/78/tox-4.23.3.dev17+g9152d396.d20241210-py3-none-any.whl
3.10: commands[0]> pytest -k test_provision_requires_ok
================================================================= test session starts =================================================================
platform darwin -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0
cachedir: .tox/3.10/.pytest_cache
rootdir: /Users/ssbarnea/code/os/tox
configfile: pyproject.toml
testpaths: tests
plugins: cov-6.0.0, flaky-3.8.1, time-machine-2.16.0, devpi-server-6.14.0, anyio-4.7.0, mock-3.14.0, xdist-3.6.1
collected 1824 items / 1823 deselected / 1 selected

tests/test_provision.py E                                                                                                                       [100%]

======================================================================= ERRORS ========================================================================
____________________________________________________ ERROR at setup of test_provision_requires_ok _____________________________________________________

tox_wheel = PosixPath('/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105/dist0/tox-4.23.4-py3-none-any.whl')
tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x101e226b0>, _basetemp=PosixPath...rs/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105'), _retention_count=3, _retention_policy='failed')

    @pytest.fixture(scope="session")
    def tox_wheels(tox_wheel: Path, tmp_path_factory: TempPathFactory) -> list[Path]:
        with elapsed("acquire dependencies for current tox"):  # takes around 1.5s if already cached
            result: list[Path] = [tox_wheel]
            info = tmp_path_factory.mktemp("info")
            with ZipFile(str(tox_wheel), "r") as zip_file:
                zip_file.extractall(path=info)
            dist_info = next((i for i in info.iterdir() if i.suffix == ".dist-info"), None)
            if dist_info is None:  # pragma: no cover
                msg = f"no tox.dist-info inside {tox_wheel}"
                raise RuntimeError(msg)
            distribution = Distribution.at(dist_info)
            wheel_cache = ROOT / ".wheel_cache" / f"{sys.version_info.major}.{sys.version_info.minor}"
            wheel_cache.mkdir(parents=True, exist_ok=True)
            cmd = [sys.executable, "-I", "-m", "pip", "download", "-d", str(wheel_cache)]
            assert distribution.requires is not None
            for req in distribution.requires:
                requirement = Requirement(req)
                if not requirement.extras:  # pragma: no branch  # we don't need to install any extras (tests/docs/etc)
                    cmd.append(req)
>           check_call(cmd)

cmd        = ['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', ...]
dist_info  = PosixPath('/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105/info0/tox-4.23.4.dist-info')
distribution = <importlib.metadata.PathDistribution object at 0x1046ae8c0>
info       = PosixPath('/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105/info0')
req        = "pytest>=8.3.3; extra == 'test'"
requirement = <Requirement('pytest>=8.3.3; extra == "test"')>
result     = [PosixPath('/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105/dist0/tox-4.23.4-py3-none-any.whl')]
tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x101e226b0>, _basetemp=PosixPath...rs/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105'), _retention_count=3, _retention_policy='failed')
tox_wheel  = PosixPath('/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pytest-of-ssbarnea/pytest-105/dist0/tox-4.23.4-py3-none-any.whl')
wheel_cache = PosixPath('/Users/ssbarnea/code/os/tox/.wheel_cache/3.10')
zip_file   = <zipfile.ZipFile [closed]>

tests/test_provision.py:96:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', ...],), kwargs = {}, retcode = 1
cmd = ['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', '/Users/ssbarnea/code/os/tox/.wheel_cache/3.10', 'cachetools>=5.5', 'chardet>=5.2', 'colorama>=0.4.6', 'filelock>=3.16.1', 'packaging>=24.2', 'platformdirs>=4.3.6', 'pluggy>=1.5', 'pyproject-api>=1.8', "tomli>=2.1; python_version < '3.11'", "typing-extensions>=4.12.2; python_version < '3.11'", 'virtualenv>=20.27.1', "devpi-process>=1.0.2; extra == 'test'", "pytest-mock>=3.14; extra == 'test'", "pytest>=8.3.3; extra == 'test'"]' returned non-zero exit status 1.

cmd        = ['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', ...]
kwargs     = {}
popenargs  = (['/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3', '-I', '-m', 'pip', 'download', '-d', ...],)
retcode    = 1

/opt/homebrew/Cellar/python@3.10/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py:369: CalledProcessError
---------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------
done in 0.43799304217100143s acquire current tox wheel
done in 0.057830207981169224s acquire dependencies for current tox
---------------------------------------------------------------- Captured stderr setup ----------------------------------------------------------------
/Users/ssbarnea/code/os/tox/.tox/3.10/bin/python3: No module named pip

Minimal example

[gaborbernat]

tox-uv is not supported by this project knowingly

[ssbarnea]

@gaborbernat That issue was raised inside tox-uv project, not tox. Closing is as not supported here is bit unexpected... as it would make impossible to properly test tox-uv if we cannot find a way test tox itself with it, right?

I can understand why you might want to avoid the noise related to uv in tox itself, but the reality is that right now, if you happen to have the tox-uv installed, you can no longer run tox own tests even if you use tox to run them. (no isolation). I found this myself while trying to make other changes to tox. To clarify, tox tests are starting and most of them are passing, just less than are failing if you happen to have tox-uv installed.

My first pain point now is to find a way to contribute to tox without having to uninstall tox-uv at user level (as apparently it works with most python project I tested so far).

Any suggestions?

[gaborbernat]

My first pain point now is to find a way to contribute to tox without having to uninstall tox-uv at user level (as apparently it works with most python project I tested so far).

Any suggestions?

tox r - e dev
.tox/dev/bin/tox r -e 3.13

[ssbarnea]

I guess that the only way to address this is to add a feature to tox itself similar to the one in pytest, where you can disable/ban a plugin like -p no:tox-uv.