401 when running Gemini on VertexAI in the Cloud

[vricciardulli]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI

Description

I'm running Gemini 1.5 with VertexAI provider in the Cloud (GCP) and after some succcesful requests I get 401 for some time with response:

{
    "error": {
        "code": 401,
        "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
        "status": "UNAUTHENTICATED",
        "details": [
            {
                "@type": "type.googleapis.com/google.rpc.ErrorInfo",
                "reason": "ACCESS_TOKEN_EXPIRED",
                "domain": "googleapis.com",
                "metadata": {
                    "service": "aiplatform.googleapis.com",
                    "method": "google.cloud.aiplatform.v1.PredictionService.GenerateContent"
                }
            }
        ]
    }
}

I see in the code that PydanticAI uses itw own attribute token_created to manage tokens and refreshes them after 3000s.

However, my access token is valid for 1800s only (even though GCP docs say that access tokens last 3600s...). This means that for the 20 minutes after that, all requests fail, given that the token is expired but PydanticAI doesn't know about it and so doesn't refresh it.

Wondering what was the reason for using token_created, instead of one of the properties valid or expired provided by the google-auth library to decide when to refresh?

P.s.: My service account already has the correct role to access Gemini in the Cloud.

Thank you 🙏

Example Code

# This is just a part of the code and it's not designed to do anything meaningful
# I'm actually using the async version in the Cloud
from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_vertex import GoogleVertexProvider
from pydantic_ai.settings import ModelSettings

provider = GoogleVertexProvider(project_id="...", region="...")
gemini_15 = GeminiModel("gemini-1.5-pro", provider=provider)

agent = Agent(
    gemini_15,
    result_type=str,
    model_settings=ModelSettings(temperature=0.0, max_tokens=50)
)

result = agent.run_sync("Return 1 random character from the latin alphabet.")

Python, Pydantic AI & LLM client version

python 3.12.9
pydantic-ai-slim 0.0.42
Gemini 1.5 Pro

[bigs]

Seconding this issue. I've been using the GoogleVertexProvider with a service account credential and ran into the exact same issue, with pydantic assuming the token would be valid when it wasn't, and giving the same unhelpful error. If I have time this weekend, I could look into debugging this.

[Kludex]

• Can you try this branch? Recreate access token on 401 for Google Vertex provider #1195

[Kludex]

I made a release that should fix this. Let me know if you still have issues with it. 🙏

[bigs]

@Kludex Just got around to testing this out (we had to abandon pydantic-ai for Gemini usage to keep things moving, but absolutely want to bring it back.) My issue is still persisting, which is actually a 403:

pydantic_ai.exceptions.ModelHTTPError: status_code: 403, model_name: gemini-2.0-flash-001, body: {
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.ai.generativelanguage.v1beta.GenerativeService.GenerateContent",
          "service": "generativelanguage.googleapis.com"
        }
      }
    ]
  }
}

I've confirmed that the service account and corresponding credential are valid by using the vertex python package directly. I'll keep investigating.

[bigs]

@Kludex On further investigation, when PydanticAI actually dispatches the request, it's being dispatched against the GLA APIs, not Vertex, even though I've given a GoogleVertexProvider. In retrospect, should have been obvious from the service in the error metadata, but my eyes were crossed.

After some more sleuthing, it appears as though there's some kind of singleton / global state happening. My application had three models defined at the module level:

gemini_vertex = GeminiModel(<flash on vertex api)
gemini_gla = GeminiModel(<flash on gla api>)
gemini_gla_tiny = GeminiModel(<flash tiny on gla api>)

And they were ordering sensitive. If gemini_vertex appeared last, I could import it and use it in Agents no problem. But if a GLA model came last, calls to the vertex model would try to hit the GLA API with a credential that only had vertex permissions.

My solution is to wrap instantiation of the models in a function and defer it to execution time. Maybe I was just "holding it wrong", but all the documentation pointed to this being possible. Any thoughts?

[bigs]

A little more investigation, I believe it's because of the cached HTTP client from cached_async_http_client. I think at a minimum, this cache should be scoped to provider. But, in all likelihood, it makes more sense for the client to not be cached at all. What do you think?

[Kludex]

@bigs everything you said is accurate.

• Use different HTTP clients based on providers #1242