Skip to content

Add Anthropic Claude Tokenizer Integration #8

@ivelin-web

Description

@ivelin-web
  • Install and import @anthropic-ai/tokenizer.
  • Map each Claude model (Haiku, Sonnet, Opus) to countTokens(text) calls.
  • Verify context window limits (e.g., Opus ≈ 200 k tokens).
  • Add fallback to text.length / 3.8 if the library does not recognize a new Claude model name.
  • Update tokenizers/index.ts (or equivalent) to call countTokens whenever tenant === 'claude'.
  • Test in a dummy Claude chat interface (or mock DOM) to confirm the count matches server‐side counts.

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions