pytest's tests should work without hypothesis

[blueyed]

Tests should run without hypothesis being installed, i.e. those tests should be skipped.

Currently it fails like this:

platform linux -- Python 3.8.0a2+, pytest-4.3.2.dev103+g15d60886.d20190322, py-1.8.0, pluggy-0.9.0
rootdir: …/Vcs/pytest, inifile: tox.ini
collected 2278 items / 1 errors / 2277 selected

===== ERRORS =====
_____ ERROR collecting testing/python/metafunc.py _____
ImportError while importing test module '…/Vcs/pytest/testing/python/metafunc.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
testing/python/metafunc.py:7: in <module>
    import hypothesis
E   ModuleNotFoundError: No module named 'hypothesis'
===== short test summary info =====
FAILED testing/python/metafunc.py
!!!!! Interrupted: 1 errors during collection !!!!!
===== 1 error in 4.31 seconds =====

This is an old issue, since #1470 I guess.
/cc @ceridwen

[blueyed]

I think it does not make sense to have hypothesis just for this test, and given that it did not find anything new since 2016 (although state is not saved on CI, should it maybe (if we would keep it)?), I'd rather remove it than conditionally skipping this test.

/cc @The-Compiler via #1470 (comment)

[The-Compiler]

I guess it would be good to use hypothesis in more places - what's the problem with requiring hypothesis for pytest's testsuite?

[blueyed]

What is the benefit?
Isn't it that it would become useful only when being used for longer? (i.e. it does not make sense on CI without a cache, for example)

I guess it would be good to use hypothesis in more places

Yes - I think the current single use does not warrant it really; but I have not much experience with it really.

[The-Compiler]

FWIW I've found many issues in qutebrowser on the CI without caching the database.

[nicoddemus]

I agree with @The-Compiler that we might not be using Hypothesis to its full potential, while I also see @blueyed's point that we have used it only for a single test since 2016.

How about we compromise: let's make a group effort to use Hypothesis more. After some time (say how about right after pytest 5.0 gets released) we reevaluate our usage: if we are still in the same situation, we can discuss dropping it again. How does that sound?

[blueyed]

@nicoddemus
Sounds good, except that I would drop it already for now - the patch for this is simple, and can easily be reverted then.

[nicoddemus]

Sounds good, except that I would drop it already for now - the patch for this is simple, and can easily be reverted then.

If we drop it now, we will create a small friction when we need to use it again. I would rather keep it, and follow with the plan of reviewing this after 5.0.

From my part, I don't see too much problem with just pip install hypothesis in a venv if I want to run pytest manually, or use tox.

[blueyed]

What is the friction?
Reverting a commit?
Isn't that very similar to pip install hypothesis? :)

(I am using pip install -e '.[testing]' usually, but ran into missing hypothesis often enough to report it)

[nicoddemus]

Well we can argue with it either way to be honest.

I think we should treat this as a "wake up call" for us in order to use more hypothesis. Starting it by removing the dependency doesn't seem the right way to approach this.

[blueyed]

Ok, closing this issue then - we should have a new one about using hypothesis more then.

[blueyed]

FWIW: it is causing the py38-dev build to fail: https://travis-ci.org/pytest-dev/pytest/jobs/532103482#L288