Move the Tokenizer's data into separate packages.

[tarekgh]

No description provided.

[tarekgh]

CC @ericstj @stephentoub @luisquintanilla @michaelgsharp

[codecov]

Codecov Report

Attention: Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Project coverage is 68.78%. Comparing base (be1e428) to head (9df82d6).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
test/Microsoft.ML.Tokenizers.Tests/Utils.cs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7248   +/-   ##
=======================================
  Coverage   68.77%   68.78%           
=======================================
  Files        1462     1463    +1     
  Lines      272261   272288   +27     
  Branches    28176    28177    +1     
=======================================
+ Hits       187254   187286   +32     
+ Misses      77764    77759    -5     
  Partials     7243     7243           

Flag	Coverage Δ
Debug	68.78% <97.56%> (+<0.01%)	⬆️
production	63.28% <100.00%> (+<0.01%)	⬆️
test	89.04% <96.29%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs	77.91% <100.00%> (+0.08%)	⬆️
...oft.ML.Tokenizers.Data.Tests/TokenizerDataTests.cs	100.00% <100.00%> (ø)
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs	98.98% <100.00%> (ø)
test/Microsoft.ML.Tokenizers.Tests/Utils.cs	70.00% <0.00%> (ø)

... and 2 files with indirect coverage changes

[tarekgh]

CC @LittleLittleCloud for awareness.

[ericstj]

Minor suggestion. Thank you!