You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If a developer wants to create a Tiktoken from something other than CreateByModelNameAsync, they're forced to use the Tiktoken constructor, which does synchronous I/O. There should be a factory equivalent to its constructors that use async I/O. There should likely also be a constructor that accepts the data already in memory rather than needing to load it separately.