Support additional search dirs for Fortran style `include` directives

[albertziegenhagel]

Is your feature request related to a problem? Please describe.
While it is possible to pass --include_dirs as additional search directories for pre-processor includes (#include 'file'), it seems that there is no such possibility to pass additional search directories for Fortran include statements (include 'file' without the #). My understanding is that fortls currently only searches for the include files relative to the file that includes them.

Describe the solution you'd like
It should be possible to pass additional search directories for Fortran include statements. I would suggest to just use the same --include_dirs that are used for pre-processor includes. To the best of my knowledge, this is what most (all?) compilers do: just use the paths passed via -I for both #include and include.

[gnikit]

I am not able to dive into the source code of fortls right now, but if I recall correctly I think the expectation for non-preprocessor sources is to be provided via the source_dirs option, see.

Let me know if that works, for you

[albertziegenhagel]

Unfortunately this doesn't seem to work. I think the relevant code part is

fortls/fortls/objects.py

Lines 2267 to 2270 in 351ae01

	def resolve_includes(self, workspace, path: str = None):
	file_dir = os.path.dirname(self.path)
	for inc in self.include_statements:
	file_path = os.path.normpath(os.path.join(file_dir, inc.path))

where self.path is the path of the file with the include statement and inc.path is the path used in the statement.

In the following lines it is checked that this path (file_path) is available in the workspace and then the content of it is inserted into the AST of the including file. So it is somehow related to source_dirs, since only files in one of source_dirs can be included successfully, but the lookup of the file does not actually consider all source_dirs.

Unfortunately, this implies another restriction I am facing: we have files without any extension that are included from other files. Hence, with the current implementation, these files can never be considered for includes by fortls, since there is nothing I can pass to incl_suffixes that would detect them. I guess this is related to the discussion in #300.

I would still favor using include_dirs over source_dirs as search paths for includes, for the following reasons:

• I consider include_dirs the analog to what I pass to the compilers via -I and that is where the compiler will look for both #include and include files. So, if we want fortls to behave as close to an actual compiler invocation as possible, we should mimic the same behavior.
• As a user I have more control over include_dirs than source_dirs. There might be directories that have source files in them, but those should not be considered for includes. Additionally I might be able to influence the order of directories in include_dirs (and hence the lookup priority), but this is less likely to be the case for source_dirs which are usually generated via glob patterns.
• My assumption is that include_dirs will usually contain less directories than source_dirs and hence lookup will probably be faster.
• If at some point we get to implement taking compiler options (mainly include paths and compiler definitions) from a compilation database like compile_commands.json, the include_dirs can be different per source file and I consider that "the goal", because now the editor can use the exact same information as the actual compilation. Since source_dirs is still global, it would contradict that.

To be honest, I would implement it such that if a file A is in one of source_dirs and includes a file B that was found in one of the incldue_dirs, but is not in any of source_dirs itself, this will simply add B as a source file, parse it and then include its content, regardless of the file extension of B or whatsoever (so a file that is included, does not necessarily need to be a source file on its own in the first place).

[gnikit]

The solution is to fix the source_dirs and file extensions, not to split the user interface into more options, based on the language semantics. included source are nothing special so the suggested artificial separation does not make any sense.

---

• I consider include_dirs the analog to what I pass to the compilers via -I and that is where the compiler will look for both #include and include files. So, if we want fortls to behave as close to an actual compiler invocation as possible, we should mimic the same behavior.

This is incorrect. -I instructs compilers where to seek for sources of AST, that includes .mod and .smod files as well, which are just vendor specific optimised AST representations of sources. source_dirs is meant to be equivalent to -I in common Fortran processors. The preprocessor in fortls is handled via other options because at the time of implementation robustness was an issue. If possible today we would merge the processor and preprocessor includes all into source_dirs, but this is a lot of work, provides no noticeable improvement over the existing UI and hence it is very low in the priority list, see #24

• As a user I have more control over include_dirs than source_dirs. There might be directories that have source files in them, but those should not be considered for includes. Additionally I might be able to influence the order of directories in include_dirs (and hence the lookup priority), but this is less likely to be the case for source_dirs which are usually generated via glob patterns.

Maybe for your specific case, but there is no compelling argument why the generic user would have more or less control over files that are inlcuded as opposed to any other Fortran source. Ordering tricks are not supported. Dependency resolution is handled internally.

• My assumption is that include_dirs will usually contain less directories than source_dirs and hence lookup will probably be faster.

Again, this is not the case for all users, also considering algorithmic performance is premature at this stage. Runtime is dominated by parsing and LSP responses not glob expansion.

• If at some point we get to implement taking compiler options (mainly include paths and compiler definitions) from a compilation database like compile_commands.json, the include_dirs can be different per source file and I consider that "the goal", because now the editor can use the exact same information as the actual compilation. Since source_dirs is still global, it would contradict that.

That is not the goal, and there is no plans to implement something like this. The serialised options you just mentioned are fully deterministic and require interacting with the build system. For multiple reasons, that wouldn't work for interactive programs like fortls. Just consider 2 cases, creating a new file, and adding modules to a file, both would be extremely hard to implement with the proposed design.

---

The proposal that code introduced via include should be treated differently, and hence deserves a separate option, than other Fortran sources is unlikely to be implemented.

[gnikit]

The way forward to resolve the issues you are having is to generalise what is treated as a source.

[gnikit]

I was playing with a solution implementation for this during the weekend. I think I will allow for full regex as source file extensions. So users will have to escape any characters that they want to be parsed as literals like dots .

[gnikit]

As mentioned, sources are controlled via the source_dirs, which include all sources of Fortran AST.
Shortcomings in the AST file extensions are addressed in PR #306.
This technically adds the ability to use REGEX as --incl_suffixes, but I have disabled it for now to not break any existing user settings. Will probably enable the option for the v3.0 release