Flake8 tests fail on CircleCI

[schlunma]

Since ~16 days, test_installation_from_source_test_mode keeps failing on CircleCI with the error:

AttributeError: module 'flake8.options.config' has no attribute 'ConfigFileFinder'

It looks like the reason is that the pytest-flake8 plugin is incompatible with flake8 >= 5.0.0 (tholo/pytest-flake8#87). At the moment no one really seems to tackle this issue, so it might be good to pin flake8 < 5.0.0 as other projects did (e.g., https://github.com/audeering/audformat/pull/257/files).

Opinions, @ESMValGroup/esmvaltool-coreteam?

[valeriupredoi]

well spotted @schlunma - see here tholo/pytest-flake8#87 - looks like we'll have to pin flake8 like others have done - EDIT: apols, I just saw you posted the same link to the issue - I can pin flake8 and open a PR right away

[valeriupredoi]

interestingly enough this issue was closed PyCQA/flake8#1652 almost as soon as it was open

[valeriupredoi]

so here's an interesting one: the GA tests that install from source (Test and Install from source eg https://github.com/ESMValGroup/ESMValCore/runs/7856342612?check_suite_focus=true incrementally downgrade flake8 to 4.0.1 that works fine - and the same thing happens when installing manually on my laptop too - gotta figure out what's the fuss in the Circle CI test (and that expkains why I didb't notice the fail until you saw it since I'm looking only at GA tests every day)

[valeriupredoi]

OK I found out - pip install ".[test]" has a slightly different behaviour than pip install -e ".[develop]" - need to dig deeper

[valeriupredoi]

interestingly enough this issue was closed PyCQA/flake8#1652 almost as soon as it was open

talking about being open and explaining things in a public GH project repo NOT 😆

[schlunma]

I agree that pip install ".[test]" and pip install -e ".[develop]" should yield the same environments, but it's probably still a good idea to pin flake8 < 5.0.0 because we should not rely on the fact that flake8 < 5.0.0 is picked by chance.

[valeriupredoi]

totally! I can do it right now, man

[schlunma]

BTW, I think what the flake8 dev means when he talks about "pytest-flake8 accesses private implementation detail" is that pytest-flake8 uses private functions of flake8 that should not be used by external code; thus, the problem needs to be fixed in pytest-flake8, not in flake8.

[valeriupredoi]

yeah I was suspecting that'd be the case - still - it's something that is more than related to them (since the issue is piped from flake8 haha), and a bit of a nicer way of commenting goes a long way, but heyho maybe he's on his last day on the job 😁