refactor(providers): consolidate Google AI Studio and Vertex AI providers with SDK alignment #6924
Conversation
Previously, authentication headers like x-goog-api-key were stripped from the cache key, allowing different API keys to share cached responses. This was a security vulnerability where User A's cached response could be served to User B if they made the same request. Changes: - Hash auth-related headers (Authorization, x-goog-api-key, x-api-key, etc.) - Include hash suffix in cache key when auth headers are present - Bump cache key version from v2 to v3 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Align authentication behavior with the official @google/genai SDK v1.34.0: Auth changes: - Centralize auth logic in new auth.ts module (GoogleAuthManager) - Gate Vertex express mode behind explicit expressMode: true flag - SDK defaults to OAuth/ADC for Vertex, not API key auth - Warn on non-SDK env vars (GEMINI_API_KEY, PALM_API_KEY, VERTEX_API_KEY) - Recommend SDK-aligned vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT) - Add mutual exclusivity check for apiKey + projectId/region API version: - Allow apiVersion override in AI Studio mode via config - Previously auto-detected based on model, now allows explicit override New modules: - auth.ts: Centralized GoogleAuthManager with API key resolution - base.ts: Shared GoogleGenericProvider base class - provider.ts: Unified provider supporting both AI Studio and Vertex - index.ts: Clean exports 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documentation updates for SDK alignment changes: Vertex docs: - Express mode requires explicit expressMode: true - Updated env vars to prioritize SDK-aligned names (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION) - Legacy vars marked as such (VERTEX_API_KEY, VERTEX_PROJECT_ID) - Add SDK Feature Comparison table showing supported/unsupported features - Document Files API, Caching API, Live API as unsupported - Point to google:image: for image generation Google AI Studio docs: - Add migration notes for SDK alignment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reverts the version bump from v2 to v3 while keeping the auth header hash suffix for security. This ensures existing cache entries remain valid. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove unused imports (GoogleAuth, parseChatPrompt, GoogleAuthManager, EnvOverrides, etc.) - Remove unused cloudLocation variable in validateAndWarn() - Rename unused parameters to underscore prefix (_context, _provider) - Add block statements for single-line if statements per Biome rules 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Express mode is now invisible to users - just provide an API key and it works automatically. No need to set `expressMode: true`. Changes: - Express mode auto-enables when API key is available in Vertex mode - Users can opt-out with `expressMode: false` if they need OAuth features - Changed apiKey+credentials warning to debug level (API key now takes precedence) - Updated tests to reflect automatic behavior - Updated vertex.md documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove explicit references to @google/genai and Python SDK - Rename "SDK Feature Comparison" → "Supported Features" - Simplify "SDK Alignment" tips → "Recommended Environment Variables" - Clean up mutual exclusivity note - Remove redundant legacy env var examples 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
mldangelo
requested review from
a team,
MrFlounder,
swarnap and
typpo
as code owners
|
⏩ No test execution environment matched (5e7061f) View output ↗ View check history
|
📝 WalkthroughWalkthroughThis pull request introduces a comprehensive refactoring of the Google AI providers architecture alongside authentication centralization and caching improvements. Documentation updates clarify environment variable precedence and authentication modes for both Google AI Studio and Vertex AI. A new centralized GoogleAuthManager module handles API key resolution from multiple environment variables with conflict detection. A shared GoogleGenericProvider base class consolidates MCP integration, tool handling, and external function loading. A unified GoogleProvider class supports both AI Studio and Vertex AI modes for Gemini models with auto-detection. The Google AI Studio provider is refactored to extend the new base and use header-based API key authentication. Vertex providers are updated similarly with simplified express-mode handling. Cache functionality is enhanced to include auth header hashing to prevent different credentials from sharing cached responses. Comprehensive test coverage validates the new auth manager, base provider, unified provider, and updated AI Studio provider. Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/providers/google/ai.studio.ts (1)
492-501: Default providers should useAIStudioChatProviderinstead ofAIStudioGenericProvider.The default grading and suggestion providers use
AIStudioGenericProvider, but this is inconsistent with the codebase pattern. Google Vertex defaults useVertexChatProvider(the specialized chat class), and all other providers (OpenAI, Mistral, GitHub, etc.) use their chat-specific implementations for defaults.AIStudioChatProviderexists in the codebase and extendsGoogleGenericProviderwith proper API endpoint handling for AI Studio—it should be used for these defaults instead of the generic provider.
🧹 Nitpick comments (11)
test/providers/google/base.test.ts (1)
104-115: Strengthen test isolation by resetting mocks, not just clearing call history
beforeEachandafterEachboth usevi.clearAllMocks(), which preserves mock implementations (important for hoisted mocks likemockMcpInstanceandMockMCPClient). To better align with the testing guidelines and avoid subtle cross‑test coupling, consider switching theafterEachtovi.resetAllMocks()so implementations are reset as well.test/providers/google/ai.studio.test.ts (1)
131-165: Endpoint and header assertions are good; consider reducing brittleness and resetting mocks
- The new tests around
getApiEndpoint,apiHost/apiBaseUrl, header‑based API key usage, andapiVersionoverride accurately reflect the refactored AI Studio behavior.- A few places hard‑code the entire URL (e.g.
'https://rendered-generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent'). That couples tests to both the default host and the Nunjucks stub; usingexpect.stringContaining('v1beta/models/gemini-pro:generateContent')plus a separate host assertion (as in other tests) would make them more resilient to future host/version tweaks.- Mocks like
mockMaybeLoadToolsFromExternalFileandtemplates.getNunjucksEngineare reconfigured in individual tests but onlyvi.clearAllMocks()is called inbeforeEach, which does not reset implementations. Adding anafterEachwithvi.resetAllMocks()(or explicitlymockReset()on the long‑lived mocks) would improve test isolation.Also applies to: 449-477, 887-930, 1032-1043, 1144-1153, 1241-1250, 1342-1355, 1425-1435
test/providers/google/auth.test.ts (1)
39-59: Tighten mock cleanup for GoogleAuthManager testsThe auth behavior tests are thorough and correctly exercise the new precedence and mutual‑exclusivity rules. To avoid any leftover mock implementations (especially for
getEnvString,logger, andmaybeLoadFromExternalFile), consider switching theafterEachcleanup fromvi.clearAllMocks()tovi.resetAllMocks(), or explicitlymockReset()the long‑lived mocks you override per test.test/providers/google/provider.test.ts (2)
97-115: Add afterEach reset for consistent mock cleanupThis suite reconfigures shared mocks (
cache.fetchWithCache,fetchWithProxy, file utilities, envars) across tests but only callsvi.clearAllMocks()inbeforeEach. Adding anafterEachwithvi.resetAllMocks()(or targetedmockReset()calls) would align with the testing guidelines and reduce the chance of state leaking between tests.
638-658: Update expressMode comment to match current auto-detect behaviorIn the express‑mode error test, the config includes
expressMode: truewith a comment saying “must be explicit”, butisExpressMode()now auto‑enables express mode whenever an API key is present andexpressModeis notfalse. The explicitexpressMode: trueis harmless but no longer required; updating the comment (or dropping the flag) would better reflect the current semantics.src/providers/google/provider.ts (1)
409-608: Response parsing covers guardrails, thinking tokens, grounding metadata, and tool callbacks
- Normalizing
GeminiApiResponseto an array, treatingpromptFeedback.blockReasonas a guardrails output (not an error), and mapping safetyfinishReasonvalues into structuredGuardrailResponseobjects aligns with the updated Vertex/AI Studio tests.- Token usage correctly tracks cached vs non‑cached responses and includes
completionDetails.reasoningwhenthoughtsTokenCountis present, matching the “thinking token” tests.- Grounding metadata extraction walks all candidates and surfaces
groundingMetadata,groundingChunks,groundingSupports, andwebSearchQueriesinmetadata, which matches the Google Search tests.- Function tool callbacks are executed only when
functionToolCallbacksis present and the output JSON contains afunctionCall, with JSON normalization ofargsbefore delegation toexecuteFunctionCallback.One small polish point: in the
blockReasonbranch,metadata.modelArmoris set toundefinedfor non‑Model Armor cases. Using a conditional spread (e.g....(isModelArmor && { modelArmor: {...} })) would avoid creating an explicitly undefined property, but this is cosmetic and doesn’t affect behavior or tests.src/providers/google/ai.studio.ts (2)
162-171: Duplicated API host resolution logic.The
getApiHost()andgetApiBaseUrl()methods share identical logic for resolving the API host from config and environment variables. This duplication could lead to maintenance issues if the precedence order changes.🔎 Consider extracting shared logic
+ private resolveApiHostFromConfig(): string | undefined { + const apiHost = + this.config.apiHost || + this.env?.GOOGLE_API_HOST || + this.env?.PALM_API_HOST || + getEnvString('GOOGLE_API_HOST') || + getEnvString('PALM_API_HOST'); + return apiHost ? getNunjucksEngine().renderString(apiHost, {}) : undefined; + } + getApiHost(): string { - const apiHost = - this.config.apiHost || - this.env?.GOOGLE_API_HOST || - this.env?.PALM_API_HOST || - getEnvString('GOOGLE_API_HOST') || - getEnvString('PALM_API_HOST') || - DEFAULT_API_HOST; - return getNunjucksEngine().renderString(apiHost, {}); + return this.resolveApiHostFromConfig() || DEFAULT_API_HOST; }Also applies to: 176-187
190-200: Non-null assertion is safe but could be clearer.The
!on line 198 is safe because the preceding condition guarantees one value is truthy. However, this pattern is somewhat fragile if the condition logic changes.🔎 Consider a clearer approach
- if ( - this.config.apiBaseUrl || - this.env?.GOOGLE_API_BASE_URL || - getEnvString('GOOGLE_API_BASE_URL') - ) { - return ( - this.config.apiBaseUrl || - this.env?.GOOGLE_API_BASE_URL || - getEnvString('GOOGLE_API_BASE_URL')! - ); - } + const apiBaseUrl = + this.config.apiBaseUrl || + this.env?.GOOGLE_API_BASE_URL || + getEnvString('GOOGLE_API_BASE_URL'); + if (apiBaseUrl) { + return apiBaseUrl; + }src/providers/google/auth.ts (3)
214-225: ADC check is synchronous buthasDefaultCredentials()is async.The
validateAndWarnmethod is synchronous, so it can't use the asynchasDefaultCredentials()method. The current approach of checkingGOOGLE_APPLICATION_CREDENTIALSdirectly is a reasonable approximation, but note that ADC can also work via metadata server (GCE/GKE) or gcloud credentials without this env var being set.Consider adding a comment explaining this limitation, or making
validateAndWarnasync in a future iteration to use the more accuratehasDefaultCredentials()check.
293-307: Credentials not validated as JSON before return.The
loadCredentialsmethod returns credentials as a raw string without validating that it's valid JSON. Invalid JSON will only fail later ingetOAuthClientat line 362, potentially with a less clear error message.🔎 Consider adding JSON validation
static loadCredentials(credentials?: string): string | undefined { if (!credentials) { return undefined; } + let result: string; if (credentials.startsWith('file://')) { try { - return maybeLoadFromExternalFile(credentials) as string; + result = maybeLoadFromExternalFile(credentials) as string; } catch (error) { throw new Error(`Failed to load credentials from file: ${error}`); } + } else { + result = credentials; } - return credentials; + // Validate JSON structure + try { + JSON.parse(result); + } catch (error) { + throw new Error(`Invalid credentials JSON: ${error}`); + } + + return result; }
398-434: Returns empty string instead of undefined when no project ID found.Line 432 returns an empty string
''as the final fallback when no project ID is resolved. This could mask configuration issues, as calling code may not distinguish between "no project configured" and "project is empty string".Consider returning
undefinedand letting callers handle the missing project ID explicitly, or throwing an error if this is a required value for Vertex mode.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (20)
site/docs/providers/google.mdsite/docs/providers/vertex.mdsrc/cache.tssrc/providers/google/ai.studio.tssrc/providers/google/auth.tssrc/providers/google/base.tssrc/providers/google/index.tssrc/providers/google/live.tssrc/providers/google/provider.tssrc/providers/google/types.tssrc/providers/google/util.tssrc/providers/google/vertex.tstest/cache.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.tstest/providers/google/base.test.tstest/providers/google/live.test.tstest/providers/google/provider.test.tstest/providers/google/util.test.tstest/providers/google/vertex.test.ts
🧰 Additional context used
📓 Path-based instructions (8)
test/**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.{ts,tsx}: Never increase test timeouts - fix the slow test instead
Never use.only()or.skip()in committed code
Always clean up mocks inafterEachusingvi.resetAllMocks()
All tests require explicit imports from vitest:import { afterEach, describe, expect, it, vi } from 'vitest'
Use Vitest's mocking utilities (vi.mock,vi.fn,vi.spyOn) and prefer shallow mocking over deep mocking
Usevi.clearAllMocks()to clear call history, but usemockReset()for full isolation when mocking withmockReturnValue()orvi.hoisted()inbeforeEach
Clean up any test data or mocks after each test
Globals are disabled in vitest configuration. All test utilities must be explicitly imported fromvitestBackend tests in
test/should use Vitest with globals enabled (describe,it,expectavailable without imports)
Files:
test/providers/google/util.test.tstest/providers/google/live.test.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use TypeScript with strict type checking
Files:
test/providers/google/util.test.tssrc/providers/google/live.tstest/providers/google/live.test.tssrc/providers/google/index.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tssrc/providers/google/types.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.tssrc/providers/google/provider.tssrc/cache.tssrc/providers/google/base.tssrc/providers/google/ai.studio.tssrc/providers/google/vertex.tssrc/providers/google/auth.tssrc/providers/google/util.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{ts,tsx,js,jsx}: Follow consistent import order (Biome handles sorting)
Use consistent curly braces for all control statements
Preferconstoverlet; avoidvar
Use object shorthand syntax whenever possible
Useasync/awaitfor asynchronous code
Use consistent error handling with proper type checks
Avoid re-exporting from files; import directly from the source module
Use the logger with object context (auto-sanitized)
Files:
test/providers/google/util.test.tssrc/providers/google/live.tstest/providers/google/live.test.tssrc/providers/google/index.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tssrc/providers/google/types.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.tssrc/providers/google/provider.tssrc/cache.tssrc/providers/google/base.tssrc/providers/google/ai.studio.tssrc/providers/google/vertex.tssrc/providers/google/auth.tssrc/providers/google/util.ts
**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.test.{ts,tsx}: Use Vitest for all tests (bothtest/andsrc/app/)
Test both success and error cases for all functionality
Files:
test/providers/google/util.test.tstest/providers/google/live.test.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
src/providers/**/*.ts
📄 CodeRabbit inference engine (src/providers/AGENTS.md)
src/providers/**/*.ts: ImplementApiProviderinterface fromsrc/types/providers.tsfor LLM provider integrations
Provider must transform prompts to provider-specific API format and return normalizedProviderResponse
If your provider allocates resources (Python workers, connections, child processes), implement acleanup()method and register withproviderRegistryfor automatic cleanup
Use logger with object context (auto-sanitized) for logging as specified indocs/agents/logging.md
OpenAI-compatible providers should extendOpenAiChatCompletionProviderbase class
Config priority order: Explicit options > Environment variables > Provider defaults
Files:
src/providers/google/live.tssrc/providers/google/index.tssrc/providers/google/types.tssrc/providers/google/provider.tssrc/providers/google/base.tssrc/providers/google/ai.studio.tssrc/providers/google/vertex.tssrc/providers/google/auth.tssrc/providers/google/util.ts
src/providers/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Document provider configurations following examples in existing code
Files:
src/providers/google/live.tssrc/providers/google/index.tssrc/providers/google/types.tssrc/providers/google/provider.tssrc/providers/google/base.tssrc/providers/google/ai.studio.tssrc/providers/google/vertex.tssrc/providers/google/auth.tssrc/providers/google/util.ts
site/docs/**/*.md
📄 CodeRabbit inference engine (site/AGENTS.md)
site/docs/**/*.md: Don't modify heading text in documentation (often externally linked)
Avoid embellishment words like 'sophisticated' in documentation
Include front matter with title (under 60 chars), description (150-160 chars), and sidebar_position on all documentation pages
Addtitle="filename.yaml"attribute only to code blocks containing complete, runnable files
Don't add titles to code block fragments (only complete, runnable files)
Use// highlight-next-linedirective to emphasize specific lines in code blocks
Never remove existing highlight directives from code blocks
Use admonition syntax (:::note, :::warning, :::danger) with empty lines around content
Files:
site/docs/providers/google.mdsite/docs/providers/vertex.md
site/**/*.{md,ts,tsx,js,json}
📄 CodeRabbit inference engine (site/AGENTS.md)
site/**/*.{md,ts,tsx,js,json}: Use 'eval' not 'evaluation' in commands and documentation
Use 'Promptfoo' when referring to the company or product, 'promptfoo' when referring to the CLI command or in code
Files:
site/docs/providers/google.mdsite/docs/providers/vertex.md
🧠 Learnings (25)
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/providers/*.test.{ts,tsx} : For provider tests, cover: success case, error cases (4xx, 5xx, rate limits), configuration validation, and token usage tracking
Applied to files:
test/providers/google/util.test.tstest/providers/google/live.test.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/test/providers/**/*.ts : Provider tests must NEVER make real API calls - mock all HTTP requests using `vi.mock`
Applied to files:
test/providers/google/util.test.tstest/providers/google/live.test.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Use `vi.clearAllMocks()` to clear call history, but use `mockReset()` for full isolation when mocking with `mockReturnValue()` or `vi.hoisted()` in `beforeEach`
Applied to files:
test/providers/google/util.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Always clean up mocks in `afterEach` using `vi.resetAllMocks()`
Applied to files:
test/providers/google/util.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-28T00:26:53.714Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/app/AGENTS.md:0-0
Timestamp: 2025-12-28T00:26:53.714Z
Learning: Applies to src/app/src/**/*.test.{ts,tsx} : Use Vitest with `vi.fn()` for mocks and `vi.mock()` for modules. See 'src/app/src/hooks/usePageMeta.test.ts' for patterns
Applied to files:
test/providers/google/util.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Use Vitest's mocking utilities (`vi.mock`, `vi.fn`, `vi.spyOn`) and prefer shallow mocking over deep mocking
Applied to files:
test/providers/google/util.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : Config priority order: Explicit options > Environment variables > Provider defaults
Applied to files:
src/providers/google/live.tstest/providers/google/base.test.tssrc/providers/google/types.tstest/providers/google/ai.studio.test.tssrc/providers/google/provider.tssite/docs/providers/vertex.mdsrc/providers/google/ai.studio.tssrc/providers/google/auth.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/test/providers/**/*.ts : Test provider success AND error cases, including rate limits, timeouts, and invalid configs
Applied to files:
test/providers/google/live.test.tstest/cache.test.tstest/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/base.test.tstest/providers/google/ai.studio.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : OpenAI-compatible providers should extend `OpenAiChatCompletionProvider` base class
Applied to files:
src/providers/google/index.tstest/providers/google/base.test.tssrc/providers/google/types.tstest/providers/google/ai.studio.test.tssrc/providers/google/provider.tssrc/providers/google/base.tssrc/providers/google/ai.studio.tssrc/providers/google/vertex.ts
📚 Learning: 2025-12-30T02:42:00.890Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-30T02:42:00.890Z
Learning: Applies to src/providers/**/*.{ts,tsx} : Document provider configurations following examples in existing code
Applied to files:
src/providers/google/index.tstest/providers/google/base.test.tssrc/providers/google/types.tssrc/providers/google/base.tssrc/providers/google/vertex.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : Implement `ApiProvider` interface from `src/types/providers.ts` for LLM provider integrations
Applied to files:
src/providers/google/index.tstest/providers/google/base.test.tssrc/providers/google/types.tssrc/providers/google/provider.tssrc/providers/google/base.ts
📚 Learning: 2025-12-20T02:05:23.936Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/redteam/AGENTS.md:0-0
Timestamp: 2025-12-20T02:05:23.936Z
Learning: Applies to src/redteam/test/redteam/**/*.{ts,js} : Add tests for new plugins in the test/redteam/ directory following the reference pattern in src/redteam/plugins/pii.ts
Applied to files:
test/cache.test.tstest/providers/google/base.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Globals are disabled in vitest configuration. All test utilities must be explicitly imported from `vitest`
Applied to files:
test/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : All tests require explicit imports from vitest: `import { afterEach, describe, expect, it, vi } from 'vitest'`
Applied to files:
test/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-30T02:42:00.890Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-30T02:42:00.890Z
Learning: Applies to test/**/*.test.{ts,tsx} : Backend tests in `test/` should use Vitest with globals enabled (`describe`, `it`, `expect` available without imports)
Applied to files:
test/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-12-30T02:42:00.890Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-30T02:42:00.890Z
Learning: Applies to src/app/**/*.test.{ts,tsx} : Frontend tests in `src/app/` should use Vitest with explicit imports
Applied to files:
test/providers/google/vertex.test.tstest/providers/google/provider.test.tstest/providers/google/auth.test.ts
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/README.md : Document any required API keys or credentials in the README
Applied to files:
site/docs/providers/google.md
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/README.md : When creating examples for specific providers, document required environment variables in the README
Applied to files:
site/docs/providers/google.md
📚 Learning: 2025-12-09T06:08:22.578Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: examples/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:22.578Z
Learning: Applies to examples/**/promptfooconfig.yaml : Use current model identifiers for OpenAI, Anthropic, and Google providers from the documentation in `site/docs/providers/`
Applied to files:
site/docs/providers/google.mdtest/providers/google/ai.studio.test.tssrc/providers/google/provider.tssite/docs/providers/vertex.mdsrc/providers/google/ai.studio.ts
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/README.md : For each environment variable, explain its purpose, how to obtain it, and any default values or constraints in the README
Applied to files:
site/docs/providers/google.md
📚 Learning: 2025-12-30T02:42:00.890Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-30T02:42:00.890Z
Learning: Applies to **/*.test.{ts,tsx} : Use Vitest for all tests (both `test/` and `src/app/`)
Applied to files:
test/providers/google/provider.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Clean up any test data or mocks after each test
Applied to files:
test/providers/google/provider.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : Provider must transform prompts to provider-specific API format and return normalized `ProviderResponse`
Applied to files:
test/providers/google/base.test.tssrc/providers/google/provider.tssrc/providers/google/ai.studio.ts
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/promptfooconfig*.yaml : Include a mix of providers when comparing model performance in configuration files
Applied to files:
site/docs/providers/vertex.md
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : If your provider allocates resources (Python workers, connections, child processes), implement a `cleanup()` method and register with `providerRegistry` for automatic cleanup
Applied to files:
src/providers/google/ai.studio.tssrc/providers/google/vertex.ts
🧬 Code graph analysis (9)
src/providers/google/live.ts (1)
src/envars.ts (1)
getEnvString(417-433)
test/cache.test.ts (1)
src/cache.ts (1)
fetchWithCache(183-298)
test/providers/google/vertex.test.ts (1)
src/providers/litellm.ts (1)
config(41-43)
test/providers/google/base.test.ts (1)
src/providers/google/auth.ts (1)
GoogleAuthManager(90-503)
test/providers/google/auth.test.ts (3)
src/providers/google/auth.ts (1)
GoogleAuthManager(90-503)src/envars.ts (1)
getEnvString(417-433)src/util/file.ts (1)
maybeLoadFromExternalFile(56-212)
src/providers/google/provider.ts (4)
src/providers/google/index.ts (2)
GoogleProvider(13-13)CompletionOptions(18-18)src/providers/google/auth.ts (3)
loadCredentials(293-307)loadCredentials(506-506)getGoogleClient(507-507)src/providers/google/util.ts (8)
loadCredentials(301-301)getGoogleClient(299-299)geminiFormatAndSystemInstructions(689-734)GeminiApiResponse(94-98)GeminiErrorResponse(60-66)GeminiResponseData(68-77)getCandidate(305-340)formatCandidateContents(342-393)src/providers/google/types.ts (1)
CompletionOptions(112-331)
src/providers/google/base.ts (3)
src/util/templates.ts (1)
getNunjucksEngine(15-55)src/providers/mcp/transform.ts (1)
transformMCPToolsToGoogle(84-116)src/providers/google/util.ts (1)
normalizeTools(419-450)
src/providers/google/ai.studio.ts (6)
src/envars.ts (1)
getEnvString(417-433)src/providers/google/base.ts (1)
GoogleProviderOptions(38-45)src/providers/google/shared.ts (1)
CHAT_MODELS(1-69)src/util/templates.ts (1)
getNunjucksEngine(15-55)src/types/providers.ts (2)
CallApiContextParams(51-82)ProviderResponse(134-196)src/cache.ts (1)
fetchWithCache(183-298)
src/providers/google/vertex.ts (6)
src/providers/google/index.ts (2)
VertexChatProvider(14-14)GoogleProviderOptions(16-16)src/providers/google/base.ts (1)
GoogleProviderOptions(38-45)src/envars.ts (1)
getEnvString(417-433)src/providers/google/auth.ts (3)
loadCredentials(293-307)loadCredentials(506-506)getGoogleClient(507-507)src/providers/google/util.ts (2)
loadCredentials(301-301)getGoogleClient(299-299)src/util/fetch/index.ts (1)
fetchWithProxy(37-140)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (25)
- GitHub Check: Test on Node 20.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 24.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 22.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 22.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 20.x and macOS-latest
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and windows-latest (shard 1/3)
- GitHub Check: Test on Node 20.x and windows-latest (shard 1/3)
- GitHub Check: Test on Node 22.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 22.x and windows-latest (shard 1/3)
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Test on Node 20.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 22.x and macOS-latest
- GitHub Check: Redteam (Staging API)
- GitHub Check: Build on Node 22.x
- GitHub Check: Redteam (Production API)
- GitHub Check: Build on Node 24.x
- GitHub Check: Share Test
- GitHub Check: Build on Node 20.x
- GitHub Check: Style Check
- GitHub Check: Build Docs
- GitHub Check: webui tests
- GitHub Check: security-scan
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (34)
site/docs/providers/google.md (1)
74-76: LGTM! Clear documentation of environment variable precedence.The admonition clearly communicates that
GOOGLE_API_KEYtakes precedence overGEMINI_API_KEY, which aligns with the SDK standardization goals of this PR.src/cache.ts (3)
145-181: Auth header hashing implementation is solid.The function properly handles different
HeadersInitformats (Headers object, array, plain object) and creates a stable hash. The 16-character SHA-256 truncation provides a good balance between cache key length and collision resistance.
217-220: Cache key auth suffix correctly prevents cross-credential pollution.The auth hash suffix ensures that requests with different authentication credentials don't share cached responses, which is critical for multi-tenant scenarios.
125-136: Auth header set is comprehensive and covers all authentication flows.Verification confirms all auth headers currently used in the codebase are included:
authorization(Bearer and Basic auth flows)x-goog-api-key(Google providers)x-api-key(remoteScoring)api-key(Azure provider)x-goog-user-project(valid Google Cloud header for proper cache differentiation)The set appropriately handles both current usage and potential future headers that affect credential differentiation.
src/providers/google/types.ts (4)
284-296: Express mode documentation is clear and SDK-aligned.The documentation explicitly states that express mode defaults to
false(uses OAuth/ADC), and must be explicitly enabled for API key authentication. This aligns with the SDK's behavior.
298-330: Auth options provide good SDK passthrough capabilities.The
googleAuthOptions,keyFilename, andscopesfields provide both advanced passthrough and convenient shorthand options. The JSDoc clearly documents their purpose and includes helpful examples.
333-382: Breaking change: strictMutualExclusivity defaults to true.The
strictMutualExclusivityfield defaults totrue, which enforces strict validation that throws errors when bothapiKeyandprojectId/regionare set. This is a breaking change as noted in the PR description.Consider:
- The default aligns with Google SDK behavior (good for SDK consistency)
- Users can set to
falsefor backward compatibility- The JSDoc clearly documents this behavior
Note: This breaking change should be prominently documented in release notes or migration guides.
1-1: google-auth-library dependency is properly declared. The package is included in package.json at version ^10.5.0, so theGoogleAuthOptionsimport is available.src/providers/google/util.ts (1)
296-303: Good refactoring: auth logic centralized with backward compatibility.Re-exporting the authentication functions from the centralized
auth.tsmodule maintains backward compatibility while improving code organization. This follows the single responsibility principle by separating auth logic from Gemini formatting utilities.site/docs/providers/vertex.md (6)
11-13: Clear provider selection guidance.The info box provides concise guidance on when to use
vertex:vsgoogle:providers, helping users make the right choice upfront.
269-293: Express mode documentation is clear and accurate.The updated documentation correctly reflects that:
- Express mode is auto-detected when an API key is present
- Only
GOOGLE_API_KEYis mentioned (aligning with standardization)- Users can opt out with
expressMode: falseif they need OAuth/ADC features
308-321: Mutual exclusivity note matches implementation.The admonition clearly documents the breaking change where API key and OAuth configurations are mutually exclusive by default, with the option to disable strict checking. This aligns with the
strictMutualExclusivityfield in types.ts.
323-353: Comprehensive advanced auth documentation.The section provides clear examples and a helpful table documenting
keyFilename,scopes, andgoogleAuthOptions. The examples show practical use cases like custom domains and proxy configuration.
361-373: Environment variables table clearly shows preference for GOOGLE_ variables.*The table properly documents:
GOOGLE_CLOUD_PROJECTas the primary project ID variableGOOGLE_CLOUD_LOCATIONfor regionGOOGLE_API_KEYfor express mode- Legacy variables marked as such with "use X instead" notes
1062-1081: Supported features table provides excellent capability overview.The table clearly indicates what's supported (✅), not supported (❌), and has caveats (
⚠️ ) with helpful notes. This makes it easy for users to understand provider capabilities at a glance.src/providers/google/base.ts (1)
79-113: LGTM! Constructor properly initializes provider with auth validation.The constructor correctly:
- Determines Vertex mode using GoogleAuthManager
- Validates auth configuration with comprehensive checks
- Defers MCP initialization to avoid blocking construction
- Supports custom provider IDs
test/providers/google/util.test.ts (1)
953-976: LGTM! Test correctly validates per-instance auth behavior.The updated test properly verifies that auth clients are created per-call rather than globally cached, aligning with the @google/genai SDK patterns. The descriptive comments clearly explain the rationale.
src/providers/google/live.ts (1)
132-135: LGTM! API key fallback properly supports both environment variables.The addition of GEMINI_API_KEY as a fallback provides backwards compatibility while maintaining the correct priority order (config.apiKey > GOOGLE_API_KEY > GEMINI_API_KEY). The comment clearly documents the Python SDK alignment.
test/providers/google/live.test.ts (1)
404-418: LGTM! Test properly handles both API key environment variables.The test correctly:
- Saves and restores both GOOGLE_API_KEY and GEMINI_API_KEY
- Deletes both before testing the error case
- Validates the updated error message
- Conditionally restores only defined values
test/cache.test.ts (1)
326-361: LGTM! Tests properly validate auth-based cache isolation.These tests correctly verify that:
- Requests with the same auth credentials hit the cache
- Requests with different auth credentials result in separate cache entries
- Both x-goog-api-key and Authorization headers are properly handled
This is a critical security enhancement to prevent cross-tenant cache pollution.
test/providers/google/vertex.test.ts (1)
101-121: GoogleAuthManager mock correctly isolates Vertex tests from real auth stateOverriding
getApiKey,determineVertexMode,resolveRegion, andresolveProjectIdwhile spreadingactual.GoogleAuthManageris a clean way to keep behavior aligned with the real implementation but avoid accidental dependence on host env/config. This should make the Vertex tests deterministic across environments.src/providers/google/index.ts (1)
1-18: Public Google provider barrel looks coherentThe index cleanly groups legacy AI Studio provider, unified
GoogleProvider, Vertex providers, base class, auth manager, and related types into a single entry point. This should simplify imports without changing behavior.src/providers/google/provider.ts (3)
72-185: API host/base URL and version resolution are consistent and configurableThe split between
getApiHost()/getApiBaseUrl()plusgetApiVersion()nicely covers:
- Vertex vs AI Studio hosts (including
region === 'global'andVERTEX_API_HOST/GOOGLE_API_HOSToverrides),- explicit
apiVersionoverrides,- auto‑detection of
v1alphafor thinking and gemini‑3 models vsv1betadefaults.The priority order (config > env overrides > defaults) matches the documented config precedence.
190-227: Auth header and express-mode logic match the new API-key semantics
getAuthHeaders()centralizesContent-Typeand mergesconfig.headers, while only attachingx-goog-api-keywhen in AI Studio or Vertex express mode. This cleanly avoids mixing API-key and OAuth flows.isExpressMode()auto‑enables express mode whenever a key is available unlessexpressMode: falseis set, which matches the PR description (automatic express with opt‑out). This is a good simplification; just be aware that a globalGOOGLE_API_KEYcan implicitly flip a Vertex config withprojectIdinto express mode ifexpressModeis not explicitly disabled.If you want to double‑check real-world behavior, it’s worth manually testing a Vertex config that has both
projectIdand an environmentGOOGLE_API_KEYto ensure the chosen mode (express vs OAuth) matches expectations in your deployment environments.
246-404: Gemini call path cleanly separates AI Studio, Vertex OAuth, and Vertex expressThe
callGeminiApiimplementation appropriately:
- Merges base and prompt config,
- Uses the shared
geminiFormatAndSystemInstructionshelper,- Pulls MCP + config tools via
getAllTools,- Applies
responseSchemawith file/JSON handling and variable substitution, and- Routes to:
- Vertex OAuth via
getClientWithCredentialsandprojects/{project}/locations/{region}URL,- Vertex express via
fetchWithProxyandx-goog-api-key,- AI Studio via
fetchWithCacheand header-based API key.Error handling distinguishes structured gaxios-style errors from generic exceptions and logs detailed error payloads only in the gaxios case, which is a reasonable balance.
src/providers/google/ai.studio.ts (1)
228-244: LGTM!The
callApimethod correctly awaits MCP initialization before proceeding, validates the API key early with a clear error message, and properly routes to Gemini or legacy PaLM paths based on model name.src/providers/google/auth.ts (3)
106-158: LGTM!The
getApiKeymethod correctly implements the config priority order (explicit options > environment variables), aligns with Python SDK patterns, and provides helpful warnings for deprecated or non-standard environment variables. The PALM_API_KEY exclusion for Vertex mode is appropriate.
318-382: LGTM!The
getOAuthClientmethod correctly handles dynamic import ofgoogle-auth-librarywith a clear error message if missing, properly merges auth options with precedence for explicit values, and gracefully handles project ID detection failures by deferring toresolveProjectId.
450-479: LGTM!The
resolveRegionmethod correctly implements SDK-aligned defaults ('global'for OAuth mode,'us-central1'for API key mode) and follows the config priority order with appropriate warnings for non-standard environment variables.src/providers/google/vertex.ts (5)
66-158: LGTM!The
VertexChatProviderclass correctly forces Vertex mode, implements proper API host resolution based on region (with special handling for'global'), and thegetAuthHeadersmethod appropriately adds the API key header only in express mode.
333-350: LGTM!The
isExpressModelogic correctly auto-enables express mode when an API key is available while allowing users to opt out withexpressMode: false. This aligns with the PR's goal of simplifying the user experience.
218-331: LGTM!The
callClaudeApimethod correctly uses OAuth authentication (no express mode for Anthropic models on Vertex) and properly constructs the rawPredict endpoint with project and location.
995-1004: LGTM!The
VertexEmbeddingProvider.getClientWithCredentialsmethod is properly updated to use the new object-basedgetGoogleClientAPI, forwarding all relevant auth configuration options consistently withVertexChatProvider.
370-388: LGTM!The Gemini API correctly uses
getAllTools(context)from the base class to consolidate MCP and config-defined tools, maintaining consistency with the AI Studio provider implementation.
| if (this.isExpressMode()) { | ||
| // Express mode: use simplified endpoint with API key | ||
| const apiKey = this.getApiKey(); | ||
| const url = `https://aiplatform.googleapis.com/${this.getApiVersion()}/publishers/${this.getPublisher()}/models/${this.modelName}:${endpoint}?key=${apiKey}`; | ||
| // Express mode: use simplified endpoint with API key in header | ||
| const url = `https://aiplatform.googleapis.com/${this.getApiVersion()}/publishers/${this.getPublisher()}/models/${this.modelName}:${endpoint}`; | ||
|
|
||
| const res = await fetchWithProxy(url, { | ||
| method: 'POST', | ||
| headers: { | ||
| 'Content-Type': 'application/json', | ||
| }, | ||
| headers: await this.getAuthHeaders(), | ||
| body: JSON.stringify(body), | ||
| signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n src/providers/google/vertex.ts | sed -n '450,500p'Repository: promptfoo/promptfoo
Length of output: 182
🏁 Script executed:
# Check the getApiHost() implementation
rg -A 5 "getApiHost\(\)" src/providers/google/vertex.tsRepository: promptfoo/promptfoo
Length of output: 1899
🏁 Script executed:
# Check if there's documentation about express mode
fd -t f -e md . | xargs rg -l -i "express.*mode" 2>/dev/null | head -20Repository: promptfoo/promptfoo
Length of output: 228
🏁 Script executed:
# Look for config.apiHost usage in the vertex provider
rg "apiHost" src/providers/google/vertex.tsRepository: promptfoo/promptfoo
Length of output: 169
🏁 Script executed:
# Check standard mode URL construction to understand the pattern
rg -B 5 -A 10 "getApiVersion\(\).*publishers" src/providers/google/vertex.tsRepository: promptfoo/promptfoo
Length of output: 3470
🏁 Script executed:
cat site/docs/providers/vertex.md | head -200Repository: promptfoo/promptfoo
Length of output: 8207
🏁 Script executed:
# Search for express mode and apiHost documentation
rg -A 10 -B 5 "express" site/docs/providers/vertex.mdRepository: promptfoo/promptfoo
Length of output: 4448
🏁 Script executed:
# Check configuration reference for apiHost documentation
rg -A 5 -B 5 "VERTEX_API_HOST|apiHost" site/docs/configuration/reference.md | head -100Repository: promptfoo/promptfoo
Length of output: 45
Express mode ignores VERTEX_API_HOST configuration.
Express mode on line 461 hardcodes aiplatform.googleapis.com instead of using getApiHost(), which means the VERTEX_API_HOST environment variable and config.apiHost overrides are silently ignored. This contradicts the documented configuration option that specifies VERTEX_API_HOST as a valid override for proxies or custom endpoints. While express mode uses a simplified global endpoint by design, users attempting to configure a proxy will find their settings unexpectedly ignored.
🤖 Prompt for AI Agents
In src/providers/google/vertex.ts around lines 459-468, express mode currently
hardcodes the host "aiplatform.googleapis.com" which ignores
VERTEX_API_HOST/config.apiHost; change the URL construction to use
this.getApiHost() (or the existing getApiVersion()/getPublisher()/modelName and
endpoint) so the host can be overridden by configuration or env var, and ensure
getAuthHeaders()/fetchWithProxy still receive the correct full URL; update any
tests or callers if they depended on the hardcoded host.
- Remove duplicate OAuthClientOptions export in auth.ts - Add ImageGenerationResponse interface in image.ts - Fix client.request type arguments (use `as` casting instead of type parameters) - Add status/statusText to FetchWithCacheResult mocks in provider.test.ts - Fix credentials type (should be JSON string, not object) - Fix parameter type 'object' -> 'OBJECT' for Gemini schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
mldangelo
force-pushed
the
feat/google-provider-refactor
branch
from
a2a42da to
8e500ce
Compare
Reverts cache.ts changes and implements auth-aware cache key discrimination directly in the Google provider layer instead. The new approach: - Adds createAuthCacheDiscriminator() helper in util.ts to hash auth headers - Passes _authHash property to fetchWithCache options (included in cache key) - Prevents cross-tenant cache pollution without modifying core cache logic This keeps the cache.ts module unchanged and contains the auth-aware caching logic within the Google provider where it's needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Updates test mocks to include createAuthCacheDiscriminator returning empty string to prevent _authHash from being added to fetchWithCache calls, keeping test assertions unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR refactors Google provider architecture to consolidate AI Studio and Vertex AI implementations, aligning with the official @google/genai SDK v1.34.0 patterns.
Key changes:
- Centralized authentication via
GoogleAuthManagerwith SDK-aligned priority order (GOOGLE_API_KEY > GEMINI_API_KEY) - New abstract
GoogleGenericProviderbase class with shared MCP integration and tool handling - Unified
GoogleProviderfor both AI Studio and Vertex AI modes - Express mode auto-detection for Vertex AI when API key is present
- Auth headers in cache key to prevent cross-tenant pollution
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/providers/google/auth.ts | New centralized auth manager handling API keys, OAuth, credentials loading |
| src/providers/google/base.ts | Abstract base class with MCP, tool normalization, function callbacks |
| src/providers/google/provider.ts | Unified provider implementation for AI Studio and Vertex modes |
| src/providers/google/vertex.ts | Simplified to extend base class, delegates auth to manager |
| src/providers/google/ai.studio.ts | Refactored to extend base class |
| src/providers/google/util.ts | Auth functions delegated to auth.ts, added cache discriminator |
| src/providers/google/types.ts | New config options for SDK alignment |
| test/providers/google/*.test.ts | Comprehensive test coverage for new architecture |
| site/docs/providers/*.md | Updated documentation for new auth patterns |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** | ||
| * Control Vertex AI express mode (API key authentication). | ||
| * Enable Vertex AI express mode (API key authentication). | ||
| * | ||
| * SDK Alignment: Express mode must be EXPLICITLY enabled. Vertex AI defaults | ||
| * to OAuth/ADC authentication. Using API keys for Vertex may hit different | ||
| * quotas or fail with project-scoped features (files, caches). | ||
| * | ||
| * Express mode is AUTOMATIC when an API key is available and no explicit | ||
| * projectId or credentials are configured. Set to `false` to force OAuth | ||
| * authentication even when an API key is present. | ||
| * Set to `true` to use API key authentication instead of OAuth/ADC. | ||
| * Requires an API key (via GOOGLE_API_KEY or config.apiKey). | ||
| * | ||
| * @default undefined (automatic detection) | ||
| * @default false (uses OAuth/ADC) | ||
| * @see https://cloud.google.com/vertex-ai/generative-ai/docs/start/express-mode |
There was a problem hiding this comment.
The description for expressMode states it must be "EXPLICITLY enabled" and defaults to false (uses OAuth/ADC). However, the implementation in isExpressMode() shows it's automatically enabled when an API key is present (unless expressMode: false). This is inconsistent with the documented behavior. The comment should reflect that express mode is AUTOMATIC when an API key is available, not that it must be explicitly enabled.
| config: { | ||
| vertexai: true, | ||
| apiKey: 'vertex-api-key', | ||
| expressMode: true, // SDK aligned: must be explicit |
There was a problem hiding this comment.
The comment states "SDK aligned: must be explicit" but the code shows expressMode: true is explicitly set. However, according to the PR description and other parts of the code, express mode should be AUTOMATIC when an API key is present. This test appears to contradict the documented behavior. Either the test should remove the explicit expressMode: true to test automatic detection, or the comment should be updated to explain why explicit mode is being tested here.
| expressMode: true, // SDK aligned: must be explicit | |
| // Explicitly enable express mode here to test this path; normally it's auto-enabled when an apiKey is present. | |
| expressMode: true, |
| /** | ||
| * Clear the cached auth client (useful for testing). | ||
| * @deprecated No longer uses global caching; each call creates a new auth instance. | ||
| */ | ||
| static clearCache(): void { | ||
| // No-op: Global caching was removed in favor of per-call auth instances | ||
| // This matches Google's official SDK behavior | ||
| } | ||
| } |
There was a problem hiding this comment.
The comment says "SDK aligned, no global cache" but this describes a significant behavior change where previously cached auth clients are now created per-call. This could have performance implications for applications making many API calls. The implementation should either document this performance impact or reconsider the no-caching approach for frequently-accessed clients.
- Fix express mode URLs in vertex.ts and provider.ts to use getApiHost() instead of hardcoded aiplatform.googleapis.com - Update expressMode JSDoc to accurately reflect auto-enable behavior when API key is present Addresses PR review comments: - CodeRabbit: Express mode now respects VERTEX_API_HOST config - Copilot: Documentation now matches implementation behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…nto feat/google-provider-refactor
- Update test comment at provider.test.ts:668 to clarify express mode is auto-enabled when apiKey is present (not "must be explicit") - Clarify auth.ts clearCache() comment to explain that token caching is handled internally by google-auth-library, not removed entirely 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Resolve conflicts: - src/providers/google/live.ts: Keep detailed OAuth2 error message from main - src/providers/google/util.ts: Keep re-exports from auth.ts, add clearCachedAuth - test/providers/google/live.test.ts: Update test to match new error message 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Summary
Major refactor of the Google provider architecture to align with the official
@google/genaiSDK v1.34.0 patterns and reduce code duplication between Google AI Studio and Vertex AI providers.Key Changes
auth.ts):GoogleAuthManagerclass handling all authentication with SDK-aligned priority orderbase.ts):GoogleGenericProviderwith shared functionality (MCP integration, tool normalization, cleanup)provider.ts):GoogleGenericGeminiProvidercombining AI Studio and Vertex AI implementationsgoogleAuthOptionspassthrough,scopes,keyFilename,strictMutualExclusivityArchitecture
SDK Alignment Details
GEMINI_API_KEYfirstGOOGLE_API_KEYfirstus-central1alwaysglobalfor OAuth modegoogleAuthOptionspassthroughapiKey+projectIdthrows errorNew Configuration Options
Breaking Changes
apiKeyANDprojectId/regionwill now throw an error by default. This matches the official SDK behavior. SetstrictMutualExclusivity: falseto restore permissive behavior.Files Changed
New Files:
src/providers/google/auth.ts- Centralized authentication manager (513 lines)src/providers/google/base.ts- Abstract base class (358 lines)src/providers/google/provider.ts- Unified provider implementation (613 lines)src/providers/google/index.ts- Clean re-exports (18 lines)Updated Files:
src/providers/google/types.ts- New config options (+100 lines)src/providers/google/ai.studio.ts- Extends new base classsrc/providers/google/vertex.ts- Simplified, uses new providersrc/providers/google/util.ts- Delegates to auth modulesrc/providers/google/live.ts- Minor updatessrc/cache.ts- Auth headers in cache key (+65 lines)Tests:
test/providers/google/auth.test.ts- Comprehensive auth tests (603 lines)test/providers/google/base.test.ts- Base class tests (486 lines)test/providers/google/provider.test.ts- Provider tests (670 lines)Documentation:
site/docs/providers/google.md- AI Studio docssite/docs/providers/vertex.md- Vertex AI docs (+117 lines)Test Plan
🤖 Generated with Claude Code