[CI Regression?] Re-enable flake8

[webknjaz]

@jaraco it looks like after #2486 flake8 is always skipped:

$ tox -- --flake8 -vv -k FLAKE8
python develop-inst-noop: ~/src/github/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+ssh://git@github.com/pypa/setuptools.git@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='14316484'
python run-test: commands[0] | pytest --flake8 -vv -k FLAKE8
================================ test session starts ================================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- ~/src/github/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: ~/src/github/pypa/setuptools, configfile: pytest.ini
plugins: checkdocs-2.1.1, shutil-1.7.0, virtualenv-1.7.0, flake8-1.0.7, black-0.3.12, cov-2.10.1, mypy-0.8.0, jaraco.test-4.0.1
collected 805 items / 690 deselected / 2 skipped / 113 selected                     

bootstrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)        [  0%]
conftest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)         [  1%]
easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)     [  2%]
[...snip...]
setuptools/installer.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8...) [ 22%]
setuptools/launch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ch...) [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 24%]
setuptools/monkey.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ch...) [ 25%]
setuptools/msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)  [ 26%]
setuptools/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 26%]
setuptools/package_index.py::FLAKE8 SKIPPED (file(s) previously passed FL...) [ 27%]
setuptools/py34compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 28%]
[...snip...]
tools/ppc64le-patch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ...) [ 99%]
tools/tox_pip.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)    [100%]

----------- coverage: platform linux, python 3.9.0-final-0 -----------
Name                                              Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------------
_distutils_hack/__init__.py                          66     43    35%   12-18, 28-33, 45-52, 63-64, 69-74, 77-88, 95-98, 105-106, 116, 120-123
_distutils_hack/override.py                           1      0   100%
[...snip...]
-------------------------------------------------------------------------------
TOTAL                                             17820  14002    21%


============================== short test summary info ==============================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
SKIPPED [115] .tox/python/lib/python3.9/site-packages/pytest_flake8.py:116: file(s) previously passed FLAKE8 checks
======================= 117 skipped, 690 deselected in 2.42s ========================
______________________________________ summary ______________________________________
  python: commands succeeded
  congratulations :)

[webknjaz]

By the way, actually invoking .tox/python/bin/python -m flake8 directly runs for 1m 24s being extremely slow to recursively scan the dirs because of https://github.com/pypa/setuptools/pull/2486/files#r546877674.

[webknjaz]

Excluding a few dirs makes flake8 finish in 16s (that's over 5x faster). Invoking flake8 revealed some other configuration problems. It makes me think that flake8 wasn't run in CI long before #2486.

[jaraco]

It's working as intended for me:

setuptools main $ git clean -fdx
...
setuptools main $ tox -- -k flake8 -v -p no:cov
python create: /Users/jaraco/code/public/pypa/setuptools/.tox/python
python develop-inst: /Users/jaraco/code/public/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+gh://pypa/setuptools@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='35910917'
python run-test: commands[0] | pytest -k flake8 -v -p no:cov
========================================================== test session starts ==========================================================
platform darwin -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/jaraco/code/public/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/setuptools, configfile: pytest.ini
plugins: flake8-1.0.7, mypy-0.8.0, black-0.3.12, jaraco.test-4.0.1, shutil-1.7.0, checkdocs-2.1.1, virtualenv-1.7.0
collected 805 items / 690 deselected / 2 skipped / 113 selected                                                                         

bootstrap.py::FLAKE8 PASSED                                                                                                       [  0%]
conftest.py::FLAKE8 PASSED                                                                                                        [  1%]
easy_install.py::FLAKE8 PASSED                                                                                                    [  2%]
pavement.py::FLAKE8 PASSED                                                                                                        [  3%]
setup.py::FLAKE8 PASSED                                                                                                           [  4%]
docs/conf.py::FLAKE8 PASSED                                                                                                       [  5%]
pkg_resources/__init__.py::FLAKE8 PASSED                                                                                          [  6%]
pkg_resources/tests/__init__.py::FLAKE8 PASSED                                                                                    [  6%]
pkg_resources/tests/test_find_distributions.py::FLAKE8 PASSED                                                                     [  7%]
pkg_resources/tests/test_markers.py::FLAKE8 PASSED                                                                                [  8%]
pkg_resources/tests/test_pkg_resources.py::FLAKE8 PASSED                                                                          [  9%]
pkg_resources/tests/test_resources.py::FLAKE8 PASSED                                                                              [ 10%]
pkg_resources/tests/test_working_set.py::FLAKE8 PASSED                                                                            [ 11%]
setuptools/__init__.py::FLAKE8 PASSED                                                                                             [ 12%]
setuptools/_deprecation_warning.py::FLAKE8 PASSED                                                                                 [ 13%]
setuptools/_imp.py::FLAKE8 PASSED                                                                                                 [ 13%]
setuptools/archive_util.py::FLAKE8 PASSED                                                                                         [ 14%]
setuptools/build_meta.py::FLAKE8 PASSED                                                                                           [ 15%]
setuptools/config.py::FLAKE8 PASSED                                                                                               [ 16%]
setuptools/dep_util.py::FLAKE8 PASSED                                                                                             [ 17%]
setuptools/depends.py::FLAKE8 PASSED                                                                                              [ 18%]
setuptools/dist.py::FLAKE8 PASSED                                                                                                 [ 19%]
setuptools/errors.py::FLAKE8 PASSED                                                                                               [ 20%]
setuptools/extension.py::FLAKE8 PASSED                                                                                            [ 20%]
setuptools/glob.py::FLAKE8 PASSED                                                                                                 [ 21%]
setuptools/installer.py::FLAKE8 PASSED                                                                                            [ 22%]
setuptools/launch.py::FLAKE8 PASSED                                                                                               [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 PASSED                                                                                           [ 24%]
setuptools/monkey.py::FLAKE8 PASSED                                                                                               [ 25%]
setuptools/msvc.py::FLAKE8 PASSED                                                                                                 [ 26%]
setuptools/namespaces.py::FLAKE8 PASSED                                                                                           [ 26%]
setuptools/package_index.py::FLAKE8 PASSED                                                                                        [ 27%]
setuptools/py34compat.py::FLAKE8 PASSED                                                                                           [ 28%]
setuptools/sandbox.py::FLAKE8 PASSED                                                                                              [ 29%]
setuptools/ssl_support.py::FLAKE8 PASSED                                                                                          [ 30%]
setuptools/unicode_utils.py::FLAKE8 PASSED                                                                                        [ 31%]
setuptools/version.py::FLAKE8 PASSED                                                                                              [ 32%]
setuptools/wheel.py::FLAKE8 PASSED                                                                                                [ 33%]
setuptools/windows_support.py::FLAKE8 PASSED                                                                                      [ 33%]
setuptools/command/__init__.py::FLAKE8 PASSED                                                                                     [ 34%]
setuptools/command/alias.py::FLAKE8 PASSED                                                                                        [ 35%]
setuptools/command/bdist_egg.py::FLAKE8 PASSED                                                                                    [ 36%]
setuptools/command/bdist_rpm.py::FLAKE8 PASSED                                                                                    [ 37%]
setuptools/command/bdist_wininst.py::FLAKE8 PASSED                                                                                [ 38%]
setuptools/command/build_clib.py::FLAKE8 PASSED                                                                                   [ 39%]
setuptools/command/build_ext.py::FLAKE8 PASSED                                                                                    [ 40%]
setuptools/command/build_py.py::FLAKE8 PASSED                                                                                     [ 40%]
setuptools/command/develop.py::FLAKE8 PASSED                                                                                      [ 41%]
setuptools/command/dist_info.py::FLAKE8 PASSED                                                                                    [ 42%]
setuptools/command/easy_install.py::FLAKE8 PASSED                                                                                 [ 43%]
setuptools/command/egg_info.py::FLAKE8 PASSED                                                                                     [ 44%]
setuptools/command/install.py::FLAKE8 PASSED                                                                                      [ 45%]
setuptools/command/install_egg_info.py::FLAKE8 PASSED                                                                             [ 46%]
setuptools/command/install_lib.py::FLAKE8 PASSED                                                                                  [ 46%]
setuptools/command/install_scripts.py::FLAKE8 PASSED                                                                              [ 47%]
setuptools/command/py36compat.py::FLAKE8 PASSED                                                                                   [ 48%]
setuptools/command/register.py::FLAKE8 PASSED                                                                                     [ 49%]
setuptools/command/rotate.py::FLAKE8 PASSED                                                                                       [ 50%]
setuptools/command/saveopts.py::FLAKE8 PASSED                                                                                     [ 51%]
setuptools/command/sdist.py::FLAKE8 PASSED                                                                                        [ 52%]
setuptools/command/setopt.py::FLAKE8 PASSED                                                                                       [ 53%]
setuptools/command/test.py::FLAKE8 PASSED                                                                                         [ 53%]
setuptools/command/upload.py::FLAKE8 PASSED                                                                                       [ 54%]
setuptools/command/upload_docs.py::FLAKE8 PASSED                                                                                  [ 55%]
setuptools/tests/__init__.py::FLAKE8 PASSED                                                                                       [ 56%]
setuptools/tests/contexts.py::FLAKE8 PASSED                                                                                       [ 57%]
setuptools/tests/environment.py::FLAKE8 PASSED                                                                                    [ 58%]
setuptools/tests/files.py::FLAKE8 PASSED                                                                                          [ 59%]
setuptools/tests/fixtures.py::FLAKE8 PASSED                                                                                       [ 60%]
setuptools/tests/namespaces.py::FLAKE8 PASSED                                                                                     [ 60%]
setuptools/tests/script-with-bom.py::FLAKE8 PASSED                                                                                [ 61%]
setuptools/tests/server.py::FLAKE8 PASSED                                                                                         [ 62%]
setuptools/tests/test_archive_util.py::FLAKE8 PASSED                                                                              [ 63%]
setuptools/tests/test_bdist_deprecations.py::FLAKE8 PASSED                                                                        [ 64%]
setuptools/tests/test_bdist_egg.py::FLAKE8 PASSED                                                                                 [ 65%]
setuptools/tests/test_build_clib.py::FLAKE8 PASSED                                                                                [ 66%]
setuptools/tests/test_build_ext.py::FLAKE8 PASSED                                                                                 [ 66%]
setuptools/tests/test_build_meta.py::FLAKE8 PASSED                                                                                [ 67%]
setuptools/tests/test_build_py.py::FLAKE8 PASSED                                                                                  [ 68%]
setuptools/tests/test_config.py::FLAKE8 PASSED                                                                                    [ 69%]
setuptools/tests/test_dep_util.py::FLAKE8 PASSED                                                                                  [ 70%]
setuptools/tests/test_depends.py::FLAKE8 PASSED                                                                                   [ 71%]
setuptools/tests/test_develop.py::FLAKE8 PASSED                                                                                   [ 72%]
setuptools/tests/test_dist.py::FLAKE8 PASSED                                                                                      [ 73%]
setuptools/tests/test_dist_info.py::FLAKE8 PASSED                                                                                 [ 73%]
setuptools/tests/test_distutils_adoption.py::FLAKE8 PASSED                                                                        [ 74%]
setuptools/tests/test_easy_install.py::FLAKE8 PASSED                                                                              [ 75%]
setuptools/tests/test_egg_info.py::FLAKE8 PASSED                                                                                  [ 76%]
setuptools/tests/test_extern.py::FLAKE8 PASSED                                                                                    [ 77%]
setuptools/tests/test_find_packages.py::FLAKE8 PASSED                                                                             [ 78%]
setuptools/tests/test_glob.py::FLAKE8 PASSED                                                                                      [ 79%]
setuptools/tests/test_install_scripts.py::FLAKE8 PASSED                                                                           [ 80%]
setuptools/tests/test_integration.py::FLAKE8 PASSED                                                                               [ 80%]
setuptools/tests/test_manifest.py::FLAKE8 PASSED                                                                                  [ 81%]
setuptools/tests/test_msvc.py::FLAKE8 PASSED                                                                                      [ 82%]
setuptools/tests/test_msvc14.py::FLAKE8 PASSED                                                                                    [ 83%]
setuptools/tests/test_namespaces.py::FLAKE8 PASSED                                                                                [ 84%]
setuptools/tests/test_packageindex.py::FLAKE8 PASSED                                                                              [ 85%]
setuptools/tests/test_register.py::FLAKE8 PASSED                                                                                  [ 86%]
setuptools/tests/test_sandbox.py::FLAKE8 PASSED                                                                                   [ 86%]
setuptools/tests/test_sdist.py::FLAKE8 PASSED                                                                                     [ 87%]
setuptools/tests/test_setopt.py::FLAKE8 PASSED                                                                                    [ 88%]
setuptools/tests/test_setuptools.py::FLAKE8 PASSED                                                                                [ 89%]
setuptools/tests/test_test.py::FLAKE8 PASSED                                                                                      [ 90%]
setuptools/tests/test_unicode_utils.py::FLAKE8 PASSED                                                                             [ 91%]
setuptools/tests/test_upload.py::FLAKE8 PASSED                                                                                    [ 92%]
setuptools/tests/test_upload_docs.py::FLAKE8 PASSED                                                                               [ 93%]
setuptools/tests/test_virtualenv.py::FLAKE8 PASSED                                                                                [ 93%]
setuptools/tests/test_wheel.py::FLAKE8 PASSED                                                                                     [ 94%]
setuptools/tests/test_windows_wrappers.py::FLAKE8 PASSED                                                                          [ 95%]
setuptools/tests/text.py::FLAKE8 PASSED                                                                                           [ 96%]
setuptools/tests/textwrap.py::FLAKE8 PASSED                                                                                       [ 97%]
tools/finalize.py::FLAKE8 PASSED                                                                                                  [ 98%]
tools/ppc64le-patch.py::FLAKE8 PASSED                                                                                             [ 99%]
tools/tox_pip.py::FLAKE8 PASSED                                                                                                   [100%]

======================================================== short test summary info ========================================================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
============================================ 115 passed, 2 skipped, 690 deselected in 12.79s ============================================
________________________________________________________________ summary ________________________________________________________________
  python: commands succeeded
  congratulations :)

setuptools main $ tox -- -k flake8 -v -p no:cov
python develop-inst-noop: /Users/jaraco/code/public/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+gh://pypa/setuptools@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='2739366957'
python run-test: commands[0] | pytest -k flake8 -v -p no:cov
========================================================== test session starts ==========================================================
platform darwin -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/jaraco/code/public/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/setuptools, configfile: pytest.ini
plugins: flake8-1.0.7, mypy-0.8.0, black-0.3.12, jaraco.test-4.0.1, shutil-1.7.0, checkdocs-2.1.1, virtualenv-1.7.0
collected 805 items / 690 deselected / 2 skipped / 113 selected                                                                         

bootstrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                            [  0%]
conftest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                             [  1%]
easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                         [  2%]
pavement.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                             [  3%]
setup.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                                [  4%]
docs/conf.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                            [  5%]
pkg_resources/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [  6%]
pkg_resources/tests/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [  6%]
pkg_resources/tests/test_find_distributions.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                          [  7%]
pkg_resources/tests/test_markers.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [  8%]
pkg_resources/tests/test_pkg_resources.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                               [  9%]
pkg_resources/tests/test_resources.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 10%]
pkg_resources/tests/test_working_set.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                 [ 11%]
setuptools/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 12%]
setuptools/_deprecation_warning.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 13%]
setuptools/_imp.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 13%]
setuptools/archive_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 14%]
setuptools/build_meta.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 15%]
setuptools/config.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 16%]
setuptools/dep_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 17%]
setuptools/depends.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 18%]
setuptools/dist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 19%]
setuptools/errors.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 20%]
setuptools/extension.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                 [ 20%]
setuptools/glob.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 21%]
setuptools/installer.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                 [ 22%]
setuptools/launch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 24%]
setuptools/monkey.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 25%]
setuptools/msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 26%]
setuptools/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 26%]
setuptools/package_index.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 27%]
setuptools/py34compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 28%]
setuptools/sandbox.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 29%]
setuptools/ssl_support.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [ 30%]
setuptools/unicode_utils.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 31%]
setuptools/version.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 32%]
setuptools/wheel.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                     [ 33%]
setuptools/windows_support.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 33%]
setuptools/command/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 34%]
setuptools/command/alias.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 35%]
setuptools/command/bdist_egg.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 36%]
setuptools/command/bdist_rpm.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 37%]
setuptools/command/bdist_wininst.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 38%]
setuptools/command/build_clib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 39%]
setuptools/command/build_ext.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 40%]
setuptools/command/build_py.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 40%]
setuptools/command/develop.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 41%]
setuptools/command/dist_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 42%]
setuptools/command/easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 43%]
setuptools/command/egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 44%]
setuptools/command/install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 45%]
setuptools/command/install_egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 46%]
setuptools/command/install_lib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 46%]
setuptools/command/install_scripts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 47%]
setuptools/command/py36compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 48%]
setuptools/command/register.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 49%]
setuptools/command/rotate.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 50%]
setuptools/command/saveopts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 51%]
setuptools/command/sdist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 52%]
setuptools/command/setopt.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 53%]
setuptools/command/test.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 53%]
setuptools/command/upload.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 54%]
setuptools/command/upload_docs.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 55%]
setuptools/tests/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 56%]
setuptools/tests/contexts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 57%]
setuptools/tests/environment.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 58%]
setuptools/tests/files.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [ 59%]
setuptools/tests/fixtures.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 60%]
setuptools/tests/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 60%]
setuptools/tests/script-with-bom.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 61%]
setuptools/tests/server.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 62%]
setuptools/tests/test_archive_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 63%]
setuptools/tests/test_bdist_deprecations.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                             [ 64%]
setuptools/tests/test_bdist_egg.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 65%]
setuptools/tests/test_build_clib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 66%]
setuptools/tests/test_build_ext.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 66%]
setuptools/tests/test_build_meta.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 67%]
setuptools/tests/test_build_py.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 68%]
setuptools/tests/test_config.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 69%]
setuptools/tests/test_dep_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 70%]
setuptools/tests/test_depends.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 71%]
setuptools/tests/test_develop.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 72%]
setuptools/tests/test_dist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 73%]
setuptools/tests/test_dist_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 73%]
setuptools/tests/test_distutils_adoption.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                             [ 74%]
setuptools/tests/test_easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 75%]
setuptools/tests/test_egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 76%]
setuptools/tests/test_extern.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 77%]
setuptools/tests/test_find_packages.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 78%]
setuptools/tests/test_glob.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 79%]
setuptools/tests/test_install_scripts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                [ 80%]
setuptools/tests/test_integration.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                    [ 80%]
setuptools/tests/test_manifest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 81%]
setuptools/tests/test_msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 82%]
setuptools/tests/test_msvc14.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 83%]
setuptools/tests/test_namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 84%]
setuptools/tests/test_packageindex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 85%]
setuptools/tests/test_register.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 86%]
setuptools/tests/test_sandbox.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 86%]
setuptools/tests/test_sdist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 87%]
setuptools/tests/test_setopt.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 88%]
setuptools/tests/test_setuptools.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 89%]
setuptools/tests/test_test.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 90%]
setuptools/tests/test_unicode_utils.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 91%]
setuptools/tests/test_upload.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 92%]
setuptools/tests/test_upload_docs.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                    [ 93%]
setuptools/tests/test_virtualenv.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 93%]
setuptools/tests/test_wheel.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 94%]
setuptools/tests/test_windows_wrappers.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                               [ 95%]
setuptools/tests/text.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 96%]
setuptools/tests/textwrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 97%]
tools/finalize.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                       [ 98%]
tools/ppc64le-patch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 99%]
tools/tox_pip.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                        [100%]

======================================================== short test summary info ========================================================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
SKIPPED [115] .tox/python/lib/python3.9/site-packages/pytest_flake8.py:116: file(s) previously passed FLAKE8 checks
================================================= 117 skipped, 690 deselected in 1.79s ==================================================
________________________________________________________________ summary ________________________________________________________________
  python: commands succeeded
  congratulations :)

Note, you don't need to pass --flake8. With the inclusion of jaraco.test and the configuration in pyproject.toml, --flake8 will be implied if the pytest flake8 plugin is present and enabled (and not if not). By default, the plugin is present and enabled, but can be disabled with -p no:flake8 or uninstalling it.

Note that I also saw the skips on the second run. That's working as intended. The flake8 tests don't need to be run a second time if the files haven't changed since they were last checked.

Also, you can see the tests ran in 13s for me, and 1.8s for the second attempt.

I did disable coverage for these runs to avoid the noisy (buggy) output that coverage generates.

[webknjaz]

Note that I also saw the skips on the second run. That's working as intended. The flake8 tests don't need to be run a second time if the files haven't changed since they were last checked.

It's interesting that I didn't see any PASSED lines and that's what I'm confused about. Still, I've found a number of misconfigured settings in the configs. Also, I think that flake8 should run w/o such problems separately because there's no other clean way of seeing what's happening when you run it.

[webknjaz]

I've made an intentional syntax error in bootstrap.py but that fails on the collection step of pytest not even getting to flake8. I guess that's fine...

[jaraco]

If you run git clean -fdx first, that should clear out the cache and I'd expect you'll get passed lines.

Agreed, it would be nice for flake8 to run naturally, though I'm okay with that usage being broken as long as the test suite is able to enforce the flake8 checks during the test run.

I believe the use of extend-ignore (jaraco/skeleton#28) should address that concern, as you describe.

[webknjaz]

I've verified that making style violations adds the failed entries and fixing that produces PASSED. I didn't realize there's some cache. Anyway, it's quite confusing to see linters among regular tests.

I believe the use of extend-ignore (jaraco/skeleton#28) should address that concern, as you describe.

Almost. Also, there's extend-exclude that still results in some violations showing up (specifically under setuptools/_distutils/).

[webknjaz]

FWIW I'll probably send a few PRs later to address what I've noticed.

[jaraco]

FWIW I'll probably send a few PRs later to address what I've noticed.

Sounds good. For this first round, I'm aiming toward simplicity and conformance with the skeleton without unduly compromising the project's expectations. I'd like to let these changes stabilize before tweaking them further. Feel free to send PRs, but I may choose to sit on them for some time, especially if they're not critical changes.

[webknjaz]

skeleton PR: jaraco/skeleton#33

[webknjaz]

setuptools PR: #2508