This is the changelog for the open source version of tiktoken.
- Improve performance by 5-20%; thank you to @nistath!
- Add
gpt-3.5-turbomodels toencoding_for_model - Add prefix matching to
encoding_for_modelto better support future model versions - Fix a bug in the README instructions on extending tiktoken
- Update the set of available encodings
- Add packaging metadata
- Add
tiktoken.encoding_for_modelto get the encoding for a specific model - Improve portability of caching logic
- Avoid use of
blobfilefor public files - Add support for Python 3.8
- Add py.typed
- Improve the public tests
- Initial release