Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
for compatibility with Python 3.10. - REL: add `pytest >= 4.6.11` to the argument `tests_require` of the function `setuptools.setup` in the script `setup.py`. This lower bound has been selected to ensure compatibility with both Python 3 and Python 2.7. See below for details. - REL: add `pytest >= 4.6.11` to file `requirements.txt` - REL: remove `nose` from argument `tests_require` of the function `setuptools.setup` in script `setup.py` - REL: remove `nose` from file `requirements.txt` - CI: update file `.travis.yml` - CI: remove the collection of coverage measurements on Travis CI, because these measurements do not collect Cython coverage with the build configuration currently used for CI testing. - DEV: remove `coveralls` from the file `requirements.txt`, because the package `coveralls` was used only on Travis CI. - TST: add a configuration file `tests/pytest.ini` and include it in `MANIFEST.in` - GIT: ignore `.pytest_cache/` in `.gitignore` ## Motivation The change from [`nose == 1.3.7`]( https://pypi.org/project/nose/1.3.7/#history) to [`pytest`]( https://pypi.org/project/pytest) is motivated by compatibility with Python 3.10. ## `nose` is incompatible with Python 3.10 The package `nose`, which was used to run the tests of `dd`, is not compatible with Python 3.10 (for details, read the last section below). Also, `nose` uses the `imp` module from CPython's standard library. The module `imp` [is deprecated]( https://docs.python.org/3.10/library/imp.html), so it may be removed in some future Python version. ## Summary of transition to `pytest` In summary, using `pytest` with the existing tests requires adding a [configuration file `tests/pytest.ini`]( https://docs.pytest.org/en/latest/reference/customize.html#configuration-file-formats) to tell `pytest` which functions, classes, and methods to collect tests from (called "discovery" of tests). The [parameter `--continue-on-collection-errors`]( https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags) tells `pytest` to not stop in case any test module fails to import, and to continue with running the tests. The ability to run the tests when some `dd` C extension modules are not installed is necessary. After transitioning the tests to `pytest`, the tests have been confirmed to run successfully: - on Python 2.7 with `pytest == 4.6.11`, and - on Python 3.9 with `pytest == 6.2.4`. ## Considering using `unittest` For writing `dd` tests, the module [`unittest`]( https://docs.python.org/3/library/unittest.html) (part of CPython's standard library) suffices. For *discovering* the tests, `unittest` seems to require that tests be methods of subclasses of the class `unittest.TestCase`. This is not the case in the tests of `dd` tests. Using `pytest` allows changing the test runner from `nosetests` with minimal changes to the tests themselves. Test discovery using `unittest` could possibly be implemented by adding a file `tests/__init__.py`, and defining in that file a function `load_tests`, following the [documentation of `unittest`]( https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover). In any case, it is simpler to use `pytest`, which requires only a configuration file. If `unittest` encounters an `ImportError` during collection of the tests (i.e., when `unittest` tries to import test modules), then it stops. There does not appear to be any way to tell `unittest` to continue and run the rest of the test modules (those that *could* be imported). ## Usage of `nose` The dependence on `nose` was minimal. Only one function was used from `nose`: the function `nose.tools.assert_raises`. The function `assert_raises` is dynamically defined [in the module `nose.tools.trivial`]( https://github.com/nose-devs/nose/blob/release_1.3.7/nose/tools/trivial.py#L32-L54) by instantiating the class `unittest.TestCase`, and setting `assert_raises` to equal the [bound method `assertRaises`]( https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises) of the instance of `TestCase`. So the function `assert_raises` from `nose` is a PEP8-compliant binding for the method `unittest.TestCase.assertRaises`. Reading the code of `unittest`: - <https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L156-L243> - <https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L704-L735> it follows that the usage: ```python with nose.tools.assert_raises(AssertionError): foo(1, 2) ``` is equivalent to the following code (`AssertionError` here is used as an example): ```python with unittest.TestCase().assertRaises(AssertionError): foo(1, 2) ``` ## Replacing usage of `nose` with `pytest` in test code The [context manager `pytest.raises`]( https://docs.pytest.org/en/latest/reference/reference.html#pytest-raises) is a [drop-in replacement]( https://en.wikipedia.org/wiki/Drop-in_replacement) for the function `nose.tools.assert_raises`: ```python with pytest.raises(AssertionError): foo(1, 2) ``` Also, the tests can still be run with `nosetests` on Python versions where `nose` is still available. ## Replacing the test runner `nosetests` with `pytest` - `pytest` correctly recognized the test files by default - `pytest` does not recognize by default methods of classes that do not start with "Test" as test methods. The configuration file is necessary for changing this behavior of `pytest` (in particular, the command-line parameter `-k` did not seem to work for classes). Relevant documentation: - <https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery> - <https://docs.pytest.org/en/latest/example/pythoncollection.html#changing-naming-conventions> - <https://docs.pytest.org/en/latest/how-to/nose.html> - <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_classes> - <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_functions> - <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_files> - The call `pytest tests/foo_test.py` imports the package `dd` from `site-packages` (assuming that the module `foo_test.py` contains the statement `import dd`). So the default behavior of `pytest` is as intended. In contrast, `nosetests tests/foo_test.py` imports the package `dd` from the local directory `dd/`, even though `dd` *is* installed under `site-packages`. In any case, `pytest` is called from within the directory `tests/`, as was done for `nosetests`. Both: - `python -m pytest tests/foo_test.py` and - `PYTHONPATH=. pytest tests/foo_test.py` result in importing `dd` from the local directory `dd/`. Relevant documentation: - <https://docs.pytest.org/en/latest/explanation/pythonpath.html#invoking-pytest-versus-python-m-pytest> - <https://docs.pytest.org/en/latest/how-to/usage.html#invoke-python> As remarked above, the parameter [`--continue-on-collection-errors`]( https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags) of `pytest` needs to be used for running the tests when some of the C extension modules are not installed. For example: ``` cd tests/ pytest -v --continue-on-collection-errors . ``` To activate [Python's development mode]( https://docs.python.org/3/library/devmode.html): ``` cd tests/ python -X dev -m pytest -vvv --continue-on-collection-errors . ``` ## Further remarks Observations about `pytest`: - detects assertions that failed, and marks their source lines - avoids the deprecated `imp` module (of the standard library) that `nose` uses (and thus the associated `DeprecationWarning`) <https://docs.python.org/3.10/library/imp.html> - running the tests of `dd` with `pytest` revealed several `DeprecationWarning`s that were previously not shown by `nose` (these warnings were about invalid escape sequences due to backslashes appearing in non-raw strings). [`pytest == 6.2.4`](https://pypi.org/project/pytest/6.2.4) is not compatible with Python 2.7. [`pytest == 4.6.11`](https://pypi.org/project/pytest/4.6.11/) is the latest version of `pytest` that is compatible with Python 2.7 (released on June 5, 2020). `pytest` specifies `python_requires` [PEP 345]( https://www.python.org/dev/peps/pep-0345/#requires-python), [PEP 503]( https://www.python.org/dev/peps/pep-0503/): - <https://github.com/pytest-dev/pytest/blob/4.6.11/setup.cfg#L48> - <https://github.com/pytest-dev/pytest/blob/5.0.0/setup.cfg#L43> So including `pytest>=4.6.11` in the file `requirements.txt` suffices to install, on each Python version, the latest version of `pytest` that is compatible with that Python version. This simplifies testing on CI, and packaging. In other words, conditional installations in the file `.travis.yml` are not needed for `pytest`, neither conditional definition of `tests_require` in the script `setup.py`. This approach leaves implicit the upper bound on `pytest` in `tests_require`. This upper bound is specified explicitly by `pytest` itself, depending on the Python version of the interpreter. It appears that `pip == 9.0.0` and `setuptools == 24.2.0` are required, to correctly implement `python_requires`: - <https://pip.pypa.io/en/stable/news/#v9-0-0> - <https://setuptools.readthedocs.io/en/latest/history.html#v24-2-1> ## How replacing usage of `nose` with `unittest` would have looked like A way to replace `nose` could have been to add a module `tests/utils.py` containing: ```python """Common functionality for tests.""" import unittest _test_case = unittest.TestCase() assert_raises = _test_case.assertRaises ``` which is close to what `nose` does. The function `assert_raises` could then be imported from the module `utils` in test modules, and used. Using `pytest` avoids the need for this workaround. ## Details about the incompatibility of `nose` with Python 3.10 [`nose == 1.3.7`]( https://pypi.org/project/nose/1.3.7/#history) imports in Python 3.10. Running `nosetests` fails, due to imports from the module `collections` of classes that have moved to the module `collections.abc`. Relevant information about CPython changes in `collections` (removal of ABCs): - <python/cpython#23754> - <https://bugs.python.org/issue37324> - <https://docs.python.org/3.10/library/collections.abc.html#collections-abstract-base-classes> - Deprecation since Python 3.3, present until Python 3.9: <https://docs.python.org/3.9/library/collections.html#module-collections> - Removed in Python 3.10: <https://docs.python.org/3.10/whatsnew/3.10.html#removed> ## About skipping tests The decorator `unittest.skipIf` is recognized by `pytest`, and skipped tests are correctly recorded and reported. In any case, note also the `pytest` test-skipping facilities: - <https://docs.pytest.org/en/latest/how-to/skipping.html> - <https://docs.pytest.org/en/latest/how-to/unittest.html> - <https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option> ## About passing `-X dev` to `python` in the `Makefile` The argument `dev` is available for the `python` option [`-X` only on Python 3.7 and higher]( https://docs.python.org/3/library/devmode.html#devmode). So the `Makefile` rules where `-X dev` appears are not compatible with ealier Python versions supported by `dd`. This is not an issue: the development environment is intended to be Python 3.9 or higher, so there is no issue with using `-X dev`. ## Avoiding interaction between tests via class attributes Avoid class attributes in test classes. Use [data attributes][1] instead. Initialize the data attributes in setup methods of the test classes, as is common practice. This approach avoids interaction (via class attributes) between test scripts that import the same modules of common tests. With `nose`, this kind of interaction apparently did not occur, as observed by test failures that were expected to happen. However, `pytest` apparently runs tests in a way that changes to imported modules (e.g., class attributes) persist between different test scripts. This `pytest` behavior was observed by the disappearance of test failures when running with `pytest` (the test failures were observable with `pytest` only when telling `pytest` to run individual test scripts, instead of collecting tests from all test scripts. The cause of the issue with `pytest` was the modification of class attributes (not [data attributes][1]) from the importing module, of classes in the imported module. The modifications were done by setting the class attribute `DD` that defines the BDD or ZDD manager class. Both the scripts `cudd_test.py` and `autoref_test.py` made modifications. The end result was `autoref` tests being run using the class `dd.cudd.BDD`. Using data attributes, instead of class attributes, and subclassing, avoids this kind of erroneous testing. This approach is explicit [PEP 20][4]. [1]: https://docs.python.org/3/tutorial/classes.html#instance-objects [2]: https://github.com/pytest-dev/pytest-xdist [3]: https://github.com/pytest-dev/pytest-forked [4]: https://www.python.org/dev/peps/pep-0020/
- Loading branch information