use of 'pytest' in setup.cfg collides with 'pytest' distutils command

[pytestbot]

Originally reported by: Jason R. Coombs (BitBucket: jaraco, GitHub: jaraco)

---

In pytest-runner 2, I adapted pytest-runner to support the 'pytest' distutils command. However, because the pytest project itself uses the [pytest] section of setup.cfg, that conflicts with the same section for the 'pytest' distutils command.

Using the 'pytest' distutils command in a project which specifies, for example, 'norecursedirs', the runner will fail because the pytest-runner distutils command doesn't recognize that parameter (or any other ini options). An example error message is:

error: error in setup.cfg: command 'PyTest' has no such option 'norecursedirs'

I see a few options here:

1. Just use the 'ptr' command name.
2. Have pytest runner distutils command disregard all unrecognized options.
3. Have pytest runner distutils command implement and ignore all of the options that might be presented in setup.cfg. It would need to do this before pytest is present (because args are processed before dependencies such as pytest are downloaded/imported).
4. Have pytest drop support for setup.cfg.
5. Require that users not use setup.cfg for pytest ini options (only support pytest.ini and tox.ini for pytest options when running under distutils).

I suspect there are other options, too. I'm not particularly happy with any of those options, but I'm leaning toward (2).

I've filed the ticket here with pytest for two reasons:

• I want the implementation to be as acceptable as possible for the pytest project to endorse it as a viable integration mechanism.
• I believe the use of the [pytest] section in the setup.cfg by the pytest library is a violation of the explicit expectation that those sections are meant for distutils commands.

@hpk42 What is your reaction? What would you suggest?

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/567

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

The reason why pytest looks in setup.cfg (as does tox) is that i'd eventually like to have all project specific metadata in a setup.cfg. I don't know how much people use a "[pytest]" section in setup.cfg these days, however. And i don't know how much of a worthwhile goal it is. Dropping support for [pytest] could only happen for the second next major release (it would be deprecated in 2.7). So i guess option "2" is the best one for you for now, indeed.

[pytestbot]

Original comment by Jason R. Coombs (BitBucket: jaraco, GitHub: jaraco):

---

I happen to use the "[pytest]" section in setup.cfg extensively, but that's just one use case. It seemed like the most canonical place to add recommended options for a project's pytest runs. Indeed, it sounds as if that would have been your recommendation as well. If the project chooses to deprecate that, it could continue to support options in the setup.cfg, but under a different section (perhaps something with spaces that would be unlikely to collide with a command). Or pytest could prefer use of one of the other files instead.

I'll explore the use of suppressing unknown options to the command (option 2). I don't know how straightforward that will be, because distutils uses its own port of getopt with which I'm not familiar. It's something I need to learn, though, to solve other setuptools issues.

[pytestbot]

Original comment by Jason R. Coombs (BitBucket: jaraco, GitHub: jaraco):

---

I looked into the code for pytest-runner and pytest, but it looks like it'll be impossible to adequately bypass this issue in pytest-runner.

First, when pytest loads settings from the .ini file, it loads the settings into a variable and references the keys throughout the code, so there's no authoritative list of options that might be expected to exist. That is, there's no way pytest-runner could introspect pytest to determine which options to whitelist, so any white list would need to be updated manually and periodically, and the list of options could be found anywhere in pytest.

Second, I looked into whitelisting any unrecognized option, but that's not possible either because pytest-runner is a distutils command, and commands are parsed in distutils.dist.Distribution, part of the stdlib. I considered adding a __getattr__ wrapper such that any option was present, like so:

diff --git a/ptr.py b/ptr.py
--- a/ptr.py
+++ b/ptr.py
@@ -6,6 +6,7 @@
 import shlex as _shlex
 import contextlib as _contextlib
 import sys as _sys
+import warnings as _warnings

 import setuptools.command.test as orig

@@ -103,3 +104,19 @@
        """
        with _save_argv(_sys.argv[:1] + self.addopts):
            self.result_code = __import__('pytest').main()
+
+   def __getattr__(self, key):
+       """
+       Because pytest (the project) was recommending or at least supporting
+       the use of setup.cfg for options to the py.test native test runner,
+       those options are now incorrectly applied to the setuptools pytest
+       command. See Pytest #567 for details.
+
+       This method works around the problem by making _any_ option appear
+       to be defined as an attribute on the command. See
+       `distutils.dist.Distribution._set_command_options` to see where
+       this workaround takes effect.
+       """
+       tmpl = "Suppressed error with option {key} to pytest command"
+       _warnings.warn(tmpl.format(**locals()))
+       return None

That doesn't work, though, because command options are not the only aspect that calls and depends on hasattr.

Third, I considered just trapping the error message, but as far as I can tell, there's no way to do that from the API exposed by distutils.

So I'm left feeling that options (2) and (3) aren't really viable without a refactoring of distutils, which is its own quagmire. And indeed, those options would only serve as a workaround and wouldn't resolve the essential conflict.

The reason why pytest looks in setup.cfg (as does tox) is that i'd eventually like to have all project specific metadata in a setup.cfg.

I see this goal in direct conflict with a 'pytest' command. Could pytest (the project) consider using a different section in setup.cfg to avoid the collision? If pytest wants to own the [pytest] section in setup.cfg, then pytest-runner should use another name for its distutils command. Assuming the current state of the art for distutils and setuptools, I don't see any other options.

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

i wonder if renaming the section to [tool:pytest] might help, in the long run

[pytestbot]

Original comment by Jason R. Coombs (BitBucket: jaraco, GitHub: jaraco):

---

Yes, something like that would work.

[RonnyPfannschmidt]

We should deprecate and drop usage in setup.cfg

[nicoddemus]

Related to #1311