Is it possible to join the implementation of GPT3 Tokenizer

[OneSeven]

Use Go to implement this function: https://platform.openai.com/tokenizer

[sashabaranov]

Related: https://github.com/openai/tiktoken

[OneSeven]

Related: https://github.com/openai/tiktoken

Thanks, but I think I need a library that can be called through golang.

[sashabaranov]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

[OneSeven]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

Do you have plans to add this functionality to the current SDK.
I would love to contribute, but my level is far from enough, sorry.

[sashabaranov]

There's no plan for that right now, but we are open for contributions 😄

I guess you can also call github.com/openai/tiktoken as a separate binary from Go.

[ealvar3z]

@OneSeven sure, I mean, we either would need to be able to embed this library (via cgo or otherwise) or would need to translate it from Rust to Go.

Isn't this library in Python? and if porting; how would you prefer the scaffolding of the porting into your repo? would it be a separate repo and then you import it into go-gpt3, etc. In other words, I am attempting to see your vision if porting it from Python to Go is feasible.

[marcel]

There's a go library already: https://github.com/samber/go-gpt-3-encoder

[OneSeven]

There's a go library already: https://github.com/samber/go-gpt-3-encoder

This library can only be used for English characters, and the correct results cannot be obtained for other languages

[sashabaranov]

@ealvar3z It's Rust wrapped in Python https://github.com/openai/tiktoken/blob/main/src/lib.rs

If it would be possible to bring tokenization with zero (or minimal) dependencies — I'm all for merging it. Otherwise, I think it makes sense to implement it in a separate repo.

[vvatanabe]

Good example of how to count tokens:

• https://github.com/pkoukk/tiktoken-go#counting-tokens-for-chat-api-calls
• https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb