Keep original line information in AST ranges, add separate mechanism for #line directives

[Martin521]

#line directives are very useful when debugging generated source code. Unfortunately, they are implemented in a way that creates problems in many other cases (like #18489, #17519, #18049, #18433, #18402). I therefore propose to implement them differently.

Currently, the lexer (tokenizer) handles the #line directives by attaching the target ranges (as defined by the #line directive arguments) to the tokens. So, all subsequent stages (parsing, type checking etc) have access only to the target ranges and no longer to the original ranges. The target ranges, however, are really needed only when formatting diagnostic messages and when writing the pdb file.

In #18049, where we needed the original ranges, we have created a work-around with "linemaps". These linemaps are created during lexing and can be used to map the target ranges back to the original ranges. This mapping is not necessarily unique, but we decided we have a good enough implementation for the typical cases like fslex/fsyacc. It is also currently coupled to the specific "nowarn" application.

I propose now to (try to) turn this around. Keep the original ranges through all stages. During lexing, create a forward map (which is unique) from the original to the target ranges. And use this map when formatting diagnostic messages and when writing the pdb. In these cases necessarily the range.FileName or range.FileIndex members are used. So we need to check all calls to these member (a few dozens in the compiler) and decide if a mapping is necessary. From a quick check I believe that we can make the map available to these places through TcConfig and TcGlobals.

I expect this to not create any changes to the fsc output. I also assume there are no consumers of the compiler service that rely on having the target ranges in the AST. For them, this would be a breaking change. They would need to apply the map when dealing with these ranges.

Feasibility can finally only be assessed by implementing this.
But if anybody sees roadblocks, please provide feedback.

[T-Gro]

I agree this would move the logic in the right direction (apply mapping selectively just in time, instead of defensively and eagerly).
The proposed approach would not be dropping information, so that each use case can decide.
The obvious disadvantage is that each use case for working with ranges will have to decide and this can introduce unintended regressions.

My main concern would be stability of debugging and related features - unfortunately this is also something where we lack appropriate test approach for end2end behavior.

I will also cc @auduchinok and @TheAngryByrd if they can think of any other critical IDE features worth thinking about.

We should also make sure we keep tools besides fslex/fsyacc in mind - https://github.com/search?q=%23line+language%3AF%23+&type=code - I think those 2 are the only ones we are dogfooding.