Skip to content

Tiktoken should support being created without synchronous I/O and with user supplied data #7008

Closed
@stephentoub

Description

@stephentoub

If a developer wants to create a Tiktoken from something other than CreateByModelNameAsync, they're forced to use the Tiktoken constructor, which does synchronous I/O. There should be a factory equivalent to its constructors that use async I/O. There should likely also be a constructor that accepts the data already in memory rather than needing to load it separately.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions