Remove test command and tests_require

[jaraco]

In this comment, @graingert proposes that we may be able to completely remove support for tests_require instead of transitioning that tooling from easy_install to pip install.

While he didn't directly propose removal, the effort would only benefit that ticket if the install functionality were completely removed, so let's explore what that will entail.

While I agree that tox is an excellent, powerful, robust solution, it's more heavy than tests_require, requiring that the user have tox installed in advance. As a more thorough solution, it also is subject to bugs and constraints that a simpler test runner is not. There are several advantages to setup.py test and tests_require over tox:

• Doesn't have any prerequisites (except the implicit one on Setuptools, which is generally assumed).
• As a result of not having any prerequisites, invocation is often a simple invocation (setup.py test) instead of multiple steps (e.g. pip install --user tox; python -m tox).
• setup.py test allows for invocation under a number of different Python versions naturally (i.e. python3.3 setup.py test or py -3.3 setup.py test) whereas tox offers "run under the python in which tox is installed" or "run for explicitly-declared python versions".
• As setup.py test doesn't use virtualenv, it's not subject to virtualenv bugs or other constraints imposed by virtualenv (such as version 14 dropping support for Python 3.2.).
• As setup.py test doesn't rely on pip, it's not subject to the bugs of pip (such as issues with --editable installs or namespace packages) or other constraints imposed by pip (such as dropping support for Python 3.2).

I consider these advantages small and easy enough to overcome, especially now that many of these issues have been resolved in setuptools, pip, and virtualenv. If we can get to a place that tox can broadly supplant the uses cases of setup.py test and pytest-runner (and thus tests_require) in practice, then yes, deprecating and removing it would be in order. Given the amount of activity and bugs I see around these tools, I'd asses they're still in active use.

Before flagging this functionality as deprecated, I'd like to survey the community about the possibility to see if there are use cases that would prove difficult to support with tox.

@graingert, would you be interested in being the champion for this effort (removing tests_require), starting with the outreach on distutils-sig and then implementing the deprecation/removal changes?

[FRidh]

Perhaps its relevant here. After discussing with @RonnyPfannschmidt we added the following to the Tox docs.

Integrating tox with setup.py test is as of October 2016 discouraged as it breaks packaging/testing approaches as used by downstream distributions which expect setup.py test to run tests with the invocation interpreter rather than setting up many virtualenvs and installing packages. If you need to define setup.py test you can better see about integrating your eventual test runner with it, here is an example of setup.py test integration with pytest. As the python eco-system rather moves away from using setup.py as a tool entry point it’s maybe best to not go for any setup.py test integration.

As a downstream we prefer to have one command that tests the package for the interpreter that is available and not any other versions.

[jaraco]

I've discovered that tox -e py or tox -e python will test against the python in which tox is installed, and in fact that's the default behavior if no envlist is provided. I use this usage by default across dozens of projects and find it quite useful.

[FRidh]

Yes, I agree, as long as no other envs are defined, then there's no problem.

[techtonik]

tox is good for CI, but too unwieldy for development. There should be an easy and intuitive way to run single test. Then this way can be combined with filesystem watcher of your choice.

[graingert]

@techtonik tox works totally fine for dev, you can use {posargs} to pass test flags to pytest: tox -e py36 -- --cov -s etc

[techtonik]

@graingert how much time does it take on you system to execute a single test with tox. On my it takes more than 5 seconds, which is not acceptable. Also, tox doesn't plan to add output redirection, which makes debugging hard.

tox-dev/tox#405
tox-dev/tox#73

[RonnyPfannschmidt]

@jaraco instead of test_requires, i'd like to see something that allows pip install -e .[test]

[mcepl]

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees. osc dependson openSUSE:Factory python-tox standard x86_64|grep python-|wc -l gives me 33 dependent packages (there are some false positives) and that seems too many for tool which should end up as a dependency of setuptools (or be on the same level in the dependency tree). We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

[pganssle]

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees.

To be clear, tox is not installed in the environment that actually runs the tests (unless you specifically want that). It manages virtual environments that have exactly the dependencies you want installed and nothing more. I suspect that this will give you fewer dependencies during the run, not more.

We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

Eliminate where or how? For your own projects or for projects you are packaging? tox is similar to make test in that each project can specify its own test commands. python -munittest discover will not work for a large number of packages, which intend for you to use pytest, and presumably also intend to have the dependencies that tox installs.