Move the Tokenizer's data into separate packages.#7248

Merged

tarekgh merged 6 commits intodotnet:mainfrom

tarekgh:SplitTokenizerDataToPackages

Oct 4, 2024

Commits on Sep 27, 2024

Move the Tokenizer's data into separate packages.
tarekgh
committed

Commits on Sep 30, 2024

Address the feedback
tarekgh
committed

Commits on Oct 1, 2024

Commits on Oct 3, 2024