-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
Description
- Install and import
@lenml/tokenizer-gemini. - If possible, use
fromPreTrained()to load the default Gemini encoder. - Map Gemini models (1.0 Pro, 1.5 Pro, Flash) to this tokenizer. Confirm max context windows (e.g., 32 k for 1.0, up to 1 M for 1.5 Pro).
- Provide a fallback: if the WASM bundle is too large or fails, default to calling Google’s REST
countTokensendpoint (requires API key) or usetext.length / 4. - Update
tokenizers/index.tsto return a function(text) => tokenizer.encode(text).lengthwhentenant === 'gemini'. - Write a simple script to feed known‐length text to the Gemini tokenizer and compare with API results for validation.
Reactions are currently unavailable