[Hinty] Type hinting: prepare tests & setup Mypy

[gpotter2]

This PR provides the tools, scripts and tests required to start "Project Hinty" = adding type hinting into Scapy.

• provides UTscapy util to use pyannotate to generate type hinting
• add Mypy tests to check regressions
• relocate test configs
• some various fixes that were failing on my test machine
• two files were used (processed) as a trial: main.py and __init__.py. Because http2.py had also been done previously, it was also included.

[codecov]

Codecov Report

Merging #2162 into master will decrease coverage by 0.8%.
The diff coverage is 59.32%.

@@            Coverage Diff             @@
##           master    #2162      +/-   ##
==========================================
- Coverage   84.78%   83.97%   -0.81%     
==========================================
  Files         202      194       -8     
  Lines       46098    44686    -1412     
==========================================
- Hits        39082    37525    -1557     
- Misses       7016     7161     +145

Impacted Files	Coverage Δ
scapy/__init__.py	84.09% <100%> (+0.36%)	⬆️
scapy/contrib/http2.py	96.79% <100%> (+1.29%)	⬆️
scapy/compat.py	79.22% <11.11%> (-20.78%)	⬇️
scapy/main.py	72.75% <73.33%> (-1.05%)	⬇️
scapy/layers/tftp.py	37.13% <0%> (-51.15%)	⬇️
scapy/arch/common.py	43.33% <0%> (-46.67%)	⬇️
scapy/layers/dhcp6.py	64.18% <0%> (-21.56%)	⬇️
scapy/arch/__init__.py	75.55% <0%> (-20%)	⬇️
scapy/supersocket.py	53.4% <0%> (-18.19%)	⬇️
scapy/consts.py	79.16% <0%> (-12.5%)	⬇️
... and 56 more

[guedou]

Really cool PR!

See my comments/remarks.

[gpotter2]

This is going to be quite some work. After having done main.py and __init__.py as a starter to test this PR, we'll probably have to do most files one by one (one per PR).

@guedou Below is a guide on how to contribute. Should we put it somewhere?

Guide on how to contribute

1. Pick a file you want to work on
2. Run the tests with UTscapy with the -x option enabled. (this requires to have pyannotate installed) This will create test/pyannotate_results
3. pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]
4. The file has been automatically processed. Now, edit it to fix the mistakes, and check your work with tox -e mypy. Remember that you can use cast(type, obj) from typing to help mypy without affecting the code.
5. If you get error: No library stub file for module ..., add an exception for it in .config/mypy/mypy.ini (follow the existing format)
6. Check that the files still pass PEP8: tox -e flake8
7. When everything passes, submit a PR. 😄

[gpotter2]

Another note: typing isn't installed on Python 2.7 (but is on Python 3.5+).
This adds a dependency to Scapy for Python 2.7. Scapy will still be able to be ran "out-of-the-box" for Python 3.5+.

Should we then choose to remove run_scapy_py2.bat ?

[guedou]

See my comments.

[guedou]

@gpotter2 the guide can be integrated into CONTRIBUTING.md

[guedou]

BTW, http2.py already uses typing.

[guedou]

BTW, can you remove line 2502 from scapy/contrib/http2.py ?

[gpotter2]

I think I've addressed all your points. I've also added http2.py to the list of files that get checked (with a few tweaks).

Also, it's up to you to merge that before 2.4.3. As we don't require typing, it shouldn't have much effect...

[gpotter2]

Should be ready to merge

[p-l-]

This is an awesome work again @gpotter2.
Could you rebase against current master?

[gpotter2]

Rebased

[p-l-]

Your call, @guedou.

[guedou]

@gpotter2 could you open an issue like #1277 to track the progress of adding types to Scapy?

[gpotter2]

This was supposed to be #2158.

[guedou]

Oh, I forgot about it ...