Closed
Description
🚀 The feature, motivation and pitch
@larryliu0820 has created a new shared repository for hosting tokenizer definitions.
The initial migration attempt was reverted in #1414 due to a tokenizer issue flagged in #1413, but should be straightforward to debug and reland
Task: Taking inspiration from #1401, reattempt this migration
Alternatives
No response
Additional context
No response
RFC (Optional)
No response