Fixes #7271 AOT for ML.Tokenizers

[euju-ms]

Fixes #7271

This PR makes ML.Tokenizers project AOT compatible.

ML.Tokenizers is made to use SourceGenerationContext for deserializing Json.

I had to create a helper class Vocabulary in order to register a JsonConverter for it.

Before the change, we have following Aot warnings on the calls to Json.Deserialize:

  Microsoft.ML.Tokenizers failed with 8 error(s) (2.6s)
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\EnglishRobertaTokenizer.cs(173,25): error IL3050: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\EnglishRobertaTokenizer.cs(173,25): error IL2026: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\BPETokenizer.cs(763,59): error IL3050: Using member 'System.Text.Json.JsonSerializer.DeserializeAsync<TValue>(Stream, JsonSerializerOptions, CancellationToken)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\BPETokenizer.cs(764,53): error IL3050: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\BPETokenizer.cs(763,59): error IL2026: Using member 'System.Text.Json.JsonSerializer.DeserializeAsync<TValue>(Stream, JsonSerializerOptions, CancellationToken)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\BPETokenizer.cs(764,53): error IL2026: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\CodeGenTokenizer.cs(1771,25): error IL3050: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.
    C:\Users\euju\RiderProjects\ml\src\Microsoft.ML.Tokenizers\Model\CodeGenTokenizer.cs(1771,25): error IL2026: Using member 'System.Text.Json.JsonSerializer.Deserialize<TValue>(Stream, JsonSerializerOptions)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.

After the change, warnings are no more :)

Note that netstandard2.0 framework is not Aot compatible as trimming is only supported for .NET 6 and later. Therefore, in order to test the compatibility for Microsoft.ML.Tokenizers project, you need to set the TargetFramework to net8.0.

Then you can add <PublishAot>true</PublishAot>.

e.g. Microsoft.ML.Tokenizers.csproj

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <IsPackable>true</IsPackable>
    <PackageDescription>Microsoft.ML.Tokenizers contains the implmentation of the tokenization used in the NLP transforms.</PackageDescription>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <PublishAot>true</PublishAot>
  </PropertyGroup>

Then building specifically for Microsoft.ML.Tokenizers should result in warnings if the project is not AOT compatible.

We are excited to review your PR.

So we can do the best job, please check:

• There's a descriptive title that will make sense to other developers some time from now.
• There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
• Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
• You have included any necessary tests in the same PR.

[euju-ms]

@dotnet-policy-service agree company="Microsoft"

[tarekgh]

@eiriktsarpalis could you please have a quick look at this change. The change is just changing the json deserialization code to use the source generator.

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Project coverage is 68.81%. Comparing base (823fc17) to head (5483e4f).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rosoft.ML.Tokenizers/Utils/StringSpanOrdinalKey.cs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7272   +/-   ##
=======================================
  Coverage   68.81%   68.81%           
=======================================
  Files        1461     1461           
  Lines      272405   272400    -5     
  Branches    28176    28176           
=======================================
  Hits       187442   187442           
+ Misses      77727    77725    -2     
+ Partials     7236     7233    -3     

Flag	Coverage Δ
Debug	68.81% <85.71%> (+<0.01%)	⬆️
production	63.30% <85.71%> (+<0.01%)	⬆️
test	89.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs	76.75% <100.00%> (-0.04%)	⬇️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs	72.76% <100.00%> (-0.03%)	⬇️
...oft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs	74.30% <100.00%> (-0.04%)	⬇️
...rosoft.ML.Tokenizers/Utils/StringSpanOrdinalKey.cs	76.08% <50.00%> (-0.51%)	⬇️

... and 3 files with indirect coverage changes

[eiriktsarpalis]

Nice 👍

[euju-ms]

Thanks for the review :)

[tarekgh]

/ba-g unrelated infrastructure failure.