Non Deterministic Vermin Runs

[ngtvspc]

Describe the bug
Subsequent runs of vermin . on the same directory have different results.

To Reproduce

1. Clone the calibre repository. git clone https://github.com/kovidgoyal/calibre
2. git checkout 7460a12a4b
3. Run vermin . at least twice and verify the results differ. I have had some subsequent runs report the same information. Here is the output from 2 subsequent runs of vermin:

(apatite) shawn@deepthink:~/.apatite/repos/calibre$ vermin .
Detecting python files..
Analyzing 1364 files using 8 processes..
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/overdrive.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/setup/win-ci.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/builtins.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/edelweiss.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/rtf2xml/options_trem.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/gui2/store/stores/chitanka_plugin.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/queue.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/urllib.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/socketserver.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/ozon.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/setup/plugins_mirror.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/http_cookie.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/setup/linux-installer.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/gui2/store/stores/beam_ebooks_de_plugin.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/devices/winusb.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/amazon.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/reprlib.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/html_entities.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/gui2/store/stores/ebookshoppe_uk_plugin.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/bypy/linux/site.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/google_images.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/douban.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/google.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/gui2/linux_file_dialogs.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/utils/winreg/lib.py
Note: Some files had incompatible versions so the results might not be correct!
Minimum required versions: 3.4
Incompatible versions:     2
(apatite) shawn@deepthink:~/.apatite/repos/calibre$ vermin .
Detecting python files..
Analyzing 1364 files using 8 processes..
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/utils/winreg/lib.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/utils/mreplace.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/setup/win-ci.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/http_cookie.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/devices/winusb.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/builtins.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/reprlib.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/odf/thumbnail.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/google_images.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/edelweiss.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/functools.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/utils/smtplib.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/srv/utils.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/queue.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/ozon.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/douban.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/overdrive.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/polyglot/socketserver.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/google.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/utils/monotonic.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/gui2/__init__.py
File with incompatible versions: /home/shawn/.apatite/repos/calibre/src/calibre/ebooks/metadata/sources/amazon.py
Note: Some files had incompatible versions so the results might not be correct!
Minimum required versions: 2.7
Incompatible versions:     3

(One thing I noticed is that the order of files analyzed seems to be different, maybe a clue? I have no idea)

Expected behavior
Subsequent runs of vermin should be consistent.

Environment (please complete the following information):
Vermin 0.6.0
Linux deepthink 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Python 3.6.8

Additional context
We found this issue while developing the version detector module for https://github.com/mahmoud/awesome-python-applications. Vermin is incredibly useful, just stumbling onto it was probably my biggest contribution to the apatite (the cli tool building the dataset for the awesome python list). We were looking at all sorts of really inaccurate methods for Python 2/3 detection previously.

[netromdk]

Thanks for noticing, @ngtvspc . The non-deterministic behavior is due to how set() doesn't yield a fixed ordering. Sorting the results will yield the same order of files to scan every time.

[mahmoud]

Nice!! :)

[ngtvspc]

Just want to comment back that I re-ran my "tests" and it seems it's resolved. Just to close the loop.

👍

[netromdk]

Thanks for getting back. 👍