Repository navigation
Conversation
Ziyann
changed the title
Open questions:
Ideally, more tests should be also added, but I wanted to wait for feedback about the approach before doing so. |
| throw new Error('Anthropic API key not provided. Please provide it again.'); | ||
| } | ||
| if (isEnabled(process.env.ANTHROPIC_USE_VERTEX)) { | ||
| let serviceKey = {}; |
There was a problem hiding this comment.
You should decouple this to a dedicated function from /packages/api
There was a problem hiding this comment.
Moved away most of it now, except the enabled check.
| "axios": "1.12.1", | ||
| "elliptic": "^6.6.1", | ||
| "form-data": "^4.0.4", | ||
| "tslib": "^2.8.1", |
There was a problem hiding this comment.
why the override? seems like this should be a dev dependency?
There was a problem hiding this comment.
As far as I see, tslib is used by google-auth-library, which isn't playing nice with version 1, that some dependencies are pulling in - at least keyv-file and cohere-ai (updated in the latest version), and possibly some AWS libraries. Without the override, I get:
TypeError: privateMap.get is not a function
at __classPrivateFieldGet (LibreChat/node_modules/@aws-crypto/crc32/node_modules/tslib/tslib.js:250:27)
at GoogleAuth.getClient (LibreChat/packages/api/dist/index.js:31449:138)
at new AnthropicVertex (LibreChat/packages/api/dist/index.js:32177:50)
|
|
||
| const mergedOptions = Object.assign(defaultOptions, options.modelOptions); | ||
|
|
||
| let creds: AnthropicCredentials = {}; |
There was a problem hiding this comment.
All the new logic as part of this should be dedicated functions related to the vertex handling.
Note, any changes here directly affect packages/api/src/endpoints/openai/config.ts and I would use that file as reference for how to structure things.
There was a problem hiding this comment.
I extracted the Vertex handling to packages/api/src/endpoints/anthropic/vertex.ts, and cleaned it up a bit, what do you think about this approach?
Ziyann
changed the title
|
@danny-avila is this just awaiting your re-review? I see this was last updated a month ago and just wanted to bump it in case it had fallen off your radar. I would very much appreciate this feature. If there's anything else that needs to be updated, I'd be happy to make some changes in support of something like this getting merged in 😁 |
|
When trying web search with Claude through VertexAI I get this response: Something went wrong.
Here's the specific error message we encountered: An error occurred while processing the request: 400
Troubleshooting URL: https://js.langchain.com/docs/troubleshooting/errors/INVALID_TOOL_RESULTS/{"type":"error","error":{"type":"invalid_request_error","message":"tools.0: Input tag 'web_search_20250305' found using 'type' does not match any of the expected tags: 'bash_20250124', 'custom', 'text_editor_20250124', 'text_editor_20250429', 'text_editor_20250728'"},"request_id":"req_vrtx_011CVKkvRzPMUqrjCUtEk9RY"} Just by setting web search on: |
For now, searching is only available as a preview/beta feature: When I developed this feature, it wasn't available at all yet, so I couldn't test it. For now, I'm not working any more on this PR, as I'm unsure if Danny will ever get to review/merge it (it's been 2 months). |
Thanks for the reference. I was able to get it working in our deployment. Would you like me to contribute back the fix to your PR? Danny takes a while to review things. I don't think he knows the relative priority of this PR to the other initiatives. |
Nice :) And yes, any help on this is welcome. |
PR here to fix merge conflicts: PR to add the web search fix: |
bb8ab78 to
e90b898
Compare
|
Alright, the PR is rebased on top of main, and searching is now supported thanks to @aron-muon. On a side note, there's an unrelated error, which should be fixed in a separate PR: The prompt-caching-2024-07-31 header has been deprecated: https://platform.claude.com/docs/en/build-with-claude/prompt-caching. This error isn't specific to Vertex AI. |
|
@danny-avila could you review this? |
1 similar comment
|
@danny-avila could you review this? |
|
is |
There was a problem hiding this comment.
Pull request overview
This PR adds support for using Anthropic models through Google Vertex AI as an alternative to the direct Anthropic API. The implementation introduces credential handling for service account keys, adds configuration options for Vertex AI region and usage, and creates a new client initialization path when Vertex AI is enabled.
Key Changes:
- Added Vertex AI client creation and credential loading utilities
- Modified credential handling to support both API keys and service account keys
- Added environment variables for Vertex AI configuration (
ANTHROPIC_USE_VERTEX,ANTHROPIC_VERTEX_REGION)
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/data-provider/src/config.ts | Added ANTHROPIC_API_KEY auth key enum value |
| packages/api/src/types/anthropic.ts | Defined AnthropicCredentials type to support both API key and service account credentials |
| packages/api/src/endpoints/anthropic/vertex.ts | Created utilities for loading Vertex AI credentials, checking credential types, and creating Vertex AI clients |
| packages/api/src/endpoints/anthropic/llm.ts | Modified getLLMConfig to accept credentials object instead of API key string, added credential parsing and Vertex AI client configuration |
| packages/api/src/endpoints/anthropic/index.ts | Exported vertex module functions |
| packages/api/package.json | Added @anthropic-ai/vertex-sdk peer dependency |
| package.json | Added tslib version override to fix dependency conflicts |
| api/server/services/Endpoints/anthropic/initialize.js | Updated initialization to conditionally load Vertex AI credentials or API key based on configuration |
| api/server/services/Config/EndpointService.js | Modified endpoint configuration to enable Anthropic when Vertex AI is configured |
| api/package.json | Added @anthropic-ai/vertex-sdk dependency |
| .env.example | Added documentation for Vertex AI configuration variables |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
| function getLLMConfig( | ||
| apiKey?: string, | ||
| options: AnthropicConfigOptions = {} as AnthropicConfigOptions, | ||
| credentials: string | AnthropicCredentials | undefined, |
There was a problem hiding this comment.
[nitpick] The parameter name 'credentials' is misleading when the function can accept a string API key. Consider renaming to 'credentialsOrApiKey' or handling the string case separately to make the API clearer. The string format appears to be for backward compatibility but creates ambiguity about the expected input format.
| const apiKey = creds[AuthKeys.ANTHROPIC_API_KEY] ?? null; | ||
|
|
||
| if (isAnthropicVertexCredentials(creds)) { | ||
| // Vertex AI configuration - use custom client | ||
| requestOptions.createClient = () => | ||
| createAnthropicVertexClient(creds, requestOptions.clientOptions); | ||
| } else if (apiKey) { |
There was a problem hiding this comment.
[nitpick] The apiKey is assigned but may remain null, which is then used in the conditional check at line 121. Consider using undefined instead of null for consistency with optional values, or use a more explicit check like creds[AuthKeys.ANTHROPIC_API_KEY] !== undefined.
| const apiKey = creds[AuthKeys.ANTHROPIC_API_KEY] ?? null; | |
| if (isAnthropicVertexCredentials(creds)) { | |
| // Vertex AI configuration - use custom client | |
| requestOptions.createClient = () => | |
| createAnthropicVertexClient(creds, requestOptions.clientOptions); | |
| } else if (apiKey) { | |
| const apiKey = creds[AuthKeys.ANTHROPIC_API_KEY]; | |
| if (isAnthropicVertexCredentials(creds)) { | |
| // Vertex AI configuration - use custom client | |
| requestOptions.createClient = () => | |
| createAnthropicVertexClient(creds, requestOptions.clientOptions); | |
| } else if (apiKey !== undefined) { |
| }); | ||
|
|
||
| return new AnthropicVertex({ | ||
| region: region, |
There was a problem hiding this comment.
Redundant property assignment. Use shorthand property syntax: region, instead of region: region,.
| region: region, | |
| region, |
|
|
||
| return new AnthropicVertex({ | ||
| region: region, | ||
| googleAuth: googleAuth, |
There was a problem hiding this comment.
Redundant property assignment. Use shorthand property syntax: googleAuth, instead of googleAuth: googleAuth,.
| googleAuth: googleAuth, | |
| googleAuth, |
What do you have in mind? It's true that setting ANTHROPIC_USE_VERTEX will cause Anthropic models to always try to use Vertex AI. Would you like to keep support for user provided Anthropic API keys? As far as I seen, there's no other method currently to configure Anthroptic models (the linked issue highlights this as well). So I'm not really sure what to change. |
|
Continued in #10780. |
Summary
This PR adds support for using Anthropic models through Vertex AI, as officially supported by the Anthropic & LangChain SDK:
https://docs.claude.com/en/api/claude-on-vertex-ai
https://js.langchain.com/docs/integrations/chat/anthropic/#custom-clients
The Vertex AI API is nearly identical to the Anthropic AI, so supporting it is relatively straightforward. Most of the changes are related to configuration and service account key handling, which I mostly based on the Google client implementation.
To use Anthropic models through Vertex AI, the
ANTHROPIC_USE_VERTEXenv var needs to be set to true. It will follow the Google service account key config (GOOGLE_SERVICE_KEY_FILE).Note that
tslibhas to overridden to version 2 to avoid this error caused by@smithy/eventstream-codec@2.2.0andcohere-aiusing a very old version of@aws-crypto/crc32, pulling in version 1:Closes #5995.
Change Type
Testing
ANTHROPIC_MODELS=claude-sonnet-4ANTHROPIC_USE_VERTEX=trueANTHROPIC_VERTEX_REGION=globalapi/data/auth.json.Test Configuration:
Checklist