Extending file preprocessing

[ShatrovOA]

Hello everyone!

This request is targeting couple issues with file preprocessing:

1. Preprocessing directive #else is not taken into consideration in case all above directives are not resolving. This issue is resolved in parse_fortran.py:2166
2. #include statement can add something new to the AST, e.g. module usage, functions, variables, etc. In such case I would like to see definitions in a hover. To resolve this issue, I added pp_adds return value to preprocess_file and FortranFile.preprocess methods. In order to resolve all additional definitions, I added a loop over new lines in FortranFile.parse method.

Some tests were also added.
@gnikit, let me know what you think
Thanks

[gnikit]

Thanks for the PR @ShatrovOA I will have a look towards the end of the week.

Preliminary thoughts based on a quick look, are that the diffs in the parser are substantial.That is a problem and it will likely hold back the PR. Good comments for your additions would go a long way to make things easier to review. I will have a more detailed look during the weekend.

[codecov]

Codecov Report

Merging #277 (7b421d2) into master (8918355) will increase coverage by 0.35%.
The diff coverage is 96.77%.

@@            Coverage Diff             @@
##           master     #277      +/-   ##
==========================================
+ Coverage   86.91%   87.27%   +0.35%     
==========================================
  Files          12       12              
  Lines        4556     4573      +17     
==========================================
+ Hits         3960     3991      +31     
+ Misses        596      582      -14     

Impacted Files	Coverage Δ
fortls/parse_fortran.py	90.53% <96.77%> (+1.43%)	⬆️

... and 3 files with indirect coverage changes

[albertziegenhagel]

I can at least confirm that the two lines added at parse_fortran.py:2166 fix #else preprocessing directives for me.

[gnikit]

Please elaborate inline and in detail as to the changes in the preprocess_file function.

The refactoring of the main parser for me is concerning.
I can't tell if there is drop of coverage in certain lines, or the base is simply out of date with the master, or there is added code (not just white space changes) and there is no 1-to-1 equivalence between the coverage of master and this PR.

Splitting changes to the parser file into multiple commits would allow for easier inspection of the changes in line by line coverage.

I will attempt to bring this PR up to date and see if there is way to add this external for-loop without causing so many diffs.

[gnikit]

I can at least confirm that the two lines added at parse_fortran.py:2166 fix #else preprocessing directives for me.

Yes, I think this PR needs to be split in 2, or at least have 2 GitHub Issues linked to it. Right now this is not a mergeable PR.

[gnikit]

hey @ShatrovOA I am really keen on merging the changes in this PR but we have to split them in 2 seperate PRs and have them accompanied by Issues.

Let's start with Issue 1. you mentioned. Let me know if you are interested in pursuing this, otherwise I will close this PR, and attempt to updated, cherry-pick and isolate the changes and myself.

[gnikit]

I am closing this PR as it appears to have been abandoned. I will try and implement certain bits of it as a separate PR. Ideally with lazy evaluations.