add fallback tokenizer

[JerryKwan]

add fallback tonenizer if tiktoken can not get encoding from model name
support llm services which provide openai compatibility api such as ollama

[lapp0]

Doesn't cl100k_base only apply to gpt4 and gpt3? Or am I missing something?

[JerryKwan]

@lapp0
As far as I know, gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large use cl100k_base, and GPT-3 models like davinci use r50k_base

[rlouf]

Please run pre-commit locally and push the changes

[lapp0]

Thanks!