WARNING:root:Warning: model not found. Using cl100k_base encoding. #38

Activity

1. areibman commented on Apr 11, 2024

   

   areibman

   on Apr 11, 2024

   Last edited by areibman

   Contributor

   More actions

   Potentially, yes. I believe Claude 1 uses cl100k, but Claude 2 and Claude 3 I'm not so sure.

   Here is the only info I've found on Claude 3: https://github.com/javirandor/anthropic-tokenizer

   And other Anthropic models:
   https://github.com/anthropics/anthropic-sdk-python/blob/bd866a1e837febd05b6f5cea322f1c42e29f2d01/src/anthropic/tokenizer.json

   For stuff like this, TokenCost will likely be a close but not exact price estimate.

   👍React with 👍2Reacted by Mo and Jihyeon Park