Tokenizer for new Gemini models

[pavelm10]

Is there a plan to add token counting support into vertexai/tokenization/_tokenizer_loading.py for new Gemini models?

• Gemini 2.0 Flash
• Gemini 2.0 Flash Lite
• Gemini 2.5 Pro

[qiaodev]

Hi @pavelm10 This repo is for Gemini 1.x models.

Gemini 2.0 tokenizer will be supported in https://github.com/googleapis/python-genai.

[ccwoolfolk]

Hi @pavelm10 This repo is for Gemini 1.x models.

Gemini 2.0 tokenizer will be supported in https://github.com/googleapis/python-genai.

@happy-qiao Thanks for the clarity on model scope for this repo. Is the 1.x limitation documented? While looking into configuration options for 2.5 flash (preview) it seems there is some general confusion about python-aiplatform vs python-genai, and it would be nice to have something concrete to point people to.

[chrisheecho]

python-genai: This is recommended and the future for gemini 2.0+ models and any Google owned model interactions. API shared with AI studio and Vertex AI

python-aiplatform: This is recommended for Vertex AI specific functionalities that do not exist in Google AI studio.

There will be some official communications coming end of this month to clarify further. thank you for your question!