Use precise ranges when requesting for semantic tokens with feature flags

[maryamariyan]

Summary of the changes

• Using feature flags

Contributes to #9092

TODO:

• Fixed/Updated tests
• Remove duplicate code

---

Benchmarks (Before & After)

• Note that the benchmarks use a fake csharp language server

BEFORE (RAN TWICE)

|                                 Method | WithMultiLineComment |      Mean |     Error |    StdDev |    Median |       Min |       Max |   Gen0 | Allocated |
|--------------------------------------- |--------------------- |----------:|----------:|----------:|----------:|----------:|----------:|-------:|----------:|
| 'Razor Semantic Tokens Range Handling' |                False |  7.155 us | 0.3139 us | 0.4601 us |  7.007 us |  6.486 us |  8.062 us |      - |   3.55 KB |
| 'Razor Semantic Tokens Range Handling' |                 True | 26.243 us | 0.5715 us | 0.8012 us | 25.995 us | 25.225 us | 28.709 us | 0.0305 |  14.49 KB |

|                                 Method | NumberOfCsSemanticRangesToReturn |       Mean |     Error |    StdDev |     Median |        Min |        Max |   Gen0 | Allocated |
|--------------------------------------- |--------------------------------- |-----------:|----------:|----------:|-----------:|-----------:|-----------:|-------:|----------:|
| 'Razor Semantic Tokens Range Endpoint' |                                0 |   4.523 us | 0.0696 us | 0.1021 us |   4.516 us |   4.325 us |   4.763 us | 0.0076 |   3.09 KB |
| 'Razor Semantic Tokens Range Endpoint' |                              100 |  10.485 us | 0.2310 us | 0.3386 us |  10.375 us |   9.853 us |  10.895 us | 0.0153 |   4.51 KB |
| 'Razor Semantic Tokens Range Endpoint' |                             1000 | 112.606 us | 1.8490 us | 2.6518 us | 111.907 us | 107.960 us | 117.205 us |      - |  11.64 KB |

|                                  Method |     Mean |   Error |   StdDev |   Median |      Min |      Max | Allocated |
|---------------------------------------- |---------:|--------:|---------:|---------:|---------:|---------:|----------:|
| 'Razor Semantic Tokens Range Scrolling' | 806.1 us | 9.94 us | 14.87 us | 805.0 us | 777.2 us | 837.8 us | 171.33 KB |

AFTER

|                                 Method | WithMultiLineComment |      Mean |     Error |    StdDev |    Median |       Min |       Max |   Gen0 | Allocated |
|--------------------------------------- |--------------------- |----------:|----------:|----------:|----------:|----------:|----------:|-------:|----------:|
| 'Razor Semantic Tokens Range Handling' |                False |  7.075 us | 0.2688 us | 0.3940 us |  7.298 us |  6.505 us |  7.555 us |      - |   3.55 KB |
| 'Razor Semantic Tokens Range Handling' |                 True | 27.461 us | 1.9639 us | 2.8787 us | 25.166 us | 24.068 us | 32.393 us | 0.0610 |  14.49 KB |


|                                 Method | NumberOfCsSemanticRangesToReturn |       Mean |     Error |    StdDev |     Median |        Min |        Max |   Gen0 | Allocated |
|--------------------------------------- |--------------------------------- |-----------:|----------:|----------:|-----------:|-----------:|-----------:|-------:|----------:|
| 'Razor Semantic Tokens Range Endpoint' |                                0 |   4.650 us | 0.0873 us | 0.1252 us |   4.641 us |   4.439 us |   5.005 us | 0.0076 |   3.09 KB |
| 'Razor Semantic Tokens Range Endpoint' |                              100 |  10.486 us | 0.2086 us | 0.2992 us |  10.515 us |   9.831 us |  10.950 us | 0.0153 |   4.45 KB |
| 'Razor Semantic Tokens Range Endpoint' |                             1000 | 112.258 us | 1.6684 us | 2.4455 us | 112.324 us | 106.607 us | 116.991 us |      - |  11.64 KB |

|                                  Method |     Mean |    Error |   StdDev |   Median |      Min |      Max | Allocated |
|---------------------------------------- |---------:|---------:|---------:|---------:|---------:|---------:|----------:|
| 'Razor Semantic Tokens Range Scrolling' | 822.8 us | 15.91 us | 23.81 us | 818.2 us | 787.1 us | 877.4 us | 171.33 KB |

[davidwengier]

Mostly looks really good, just lots of duplicated code that will be a maintenance headache, so we should unify.

[ToddGrun]

Are there any semantic tests over a string with multiple C# segments that only requests a razor span containing one of them? The diff files for such a test would nicely encompass the difference between the behavior with and without the feature flag set. #Closed

[maryamariyan]

Note for reviewers:

The PR is ready for review. It also adds new tests comparing results between when feature flag is set vs. when not. Only one odd case of a test is currently failing" GetSemanticTokens_CSharp_Nested_HTML:

    Assert.Contains() Failure
Not found: Int32[] [12, 38, 15, 18, 0]
In value:  HashSet<Int32[]> [[12, 38, 16, 18, 0], [12, 54, 1, 54, 0], [14, 0, 1, 51, 0], [14, 1, 4, 51, 0], [14, 6, 7, 51, 0], ...]

Third index for some reason is off for that test case. I'm going to take a look at that.
Aside from that, the PR should be ready.

Thanks @davidwengier and @ToddGrun for the reviews so far!

[davidwengier]

A few very small nits, but otherwise well done getting this in. Thanks for putting up with all of the comments 😁