Add AWS Bedrock support to ChatAnthropic

[stevehodgkiss]

AWS is quite a bit more involved than the Azure/OpenAI integration. It has a custom streaming protocol wrapping the underlying anthropic messages when streaming, and uses request signing rather than a simple bearer token. There's also a slight difference in the JSON payload - model name is in the query string & stream is defined by calling a different action (a different path - InvokeModel vs InvokeModelWithResponseStream). Docs.

• Updates the deps in the repo. req >= 0.5.1 is required for Bedrock support due to CanonicalURI encoding issue in aws_sigv4 wojtekmach/req#374
• Uses req's aws_sigv4 functionality to sign requests for Bedrock, rather than depend on aws-beam/ex_aws.
• BedrockConfig uses an anonymous function to obtain AWS credentials. This is because credentials are rotated frequently when running on AWS (by the instance metadata service). ExAws has good support for the instance metadata service & caching credentials from it, so applications running on AWS can do the following:

    ChatAnthropic.new!(%{
      model: "anthropic.claude-3-5-sonnet-20240620-v1:0",
      bedrock: %{
        credentials: fn ->
          config = ExAws.Config.new(:s3)
          {config.access_key_id, config.secret_access_key}
        end,
        region: "us-east-1"
      }
    })

• Adds a few modules around handling the custom application/vnd.amazon.eventstream response type when streaming.

TODO:

• Add bedrock live testing in chat_anthropic_test.exs. Used Macros to use mostly the same tests between the 2 APIs.
• Handle the various exception cases. API key issues return a 403 on AWS with a flat %{"message" => ".."} response, so this falls through to unexpected response. The stream returns empty maps with the error name as the key which need handling in do_process_response. Wondering how to handle these error cases between the stream and non-stream functions.

[brainlid]

Watching your PR, I saw your selection and use of Mimic. I researched it and was happy enough with it that I moved to it generally. That's here in PR #155. It creates a conflict that should be easily resolved.

Also, just skimming your work here, I'm somewhat shocked at how hard and complicated AWS Bedrock makes it. Yeesh!

[stevehodgkiss]

Watching your PR, I saw your selection and use of Mimic. I researched it and was happy enough with it that I moved to it generally. That's here in PR #155.

Oh, nice 👍

Also, just skimming your work here, I'm somewhat shocked at how hard and complicated AWS Bedrock makes it. Yeesh!

Yep, it's surprisingly complicated. The Azure OpenAI service looks like a very light wrapper around OpenAI's API structure for the most part, making it much easier to add support for.

I'll get back to this PR over the next week or so, but it's mostly there functionality wise. Feel free to leave any feedback on the PR as it is now, I think there's only a few changes needed around error cases & possibly some testing improvements.

I added the ability to run the same live test case/code against both anthropic and bedrock. It's convenient since both can be tested the same way, but a bit ugly and I haven't found a way to run a specific live test for only bedrock or anthropic (specifying line numbers to mix test :123 and --only live_anthropic_bedrock:true doesn't work). See the test "#{BedrockHelpers.prefix_for(api)} bits in chat anthropic test.

[Maxino22]

AWS is quite a bit more involved than the Azure/OpenAI integration. It has a custom streaming protocol wrapping the underlying anthropic messages when streaming, and uses request signing rather than a simple bearer token. There's also a slight difference in the JSON payload - model name is in the query string & stream is defined by calling a different action (a different path - InvokeModel vs InvokeModelWithResponseStream). Docs.

• Updates the deps in the repo. req >= 0.5.1 is required for Bedrock support due to CanonicalURI encoding issue in aws_sigv4 wojtekmach/req#374
• Uses req's aws_sigv4 functionality to sign requests for Bedrock, rather than depend on aws-beam/ex_aws.
• BedrockConfig uses an anonymous function to obtain AWS credentials. This is because credentials are rotated frequently when running on AWS (by the instance metadata service). ExAws has good support for the instance metadata service & caching credentials from it, so applications running on AWS can do the following:

    ChatAnthropic.new!(%{
      model: "anthropic.claude-3-5-sonnet-20240620-v1:0",
      bedrock: %{
        credentials: fn ->
          config = ExAws.Config.new(:s3)
          {config.access_key_id, config.secret_access_key}
        end,
        region: "us-east-1"
      }
    })

• Adds a few modules around handling the custom application/vnd.amazon.eventstream response type when streaming.

TODO:

• Add bedrock live testing in chat_anthropic_test.exs. Used Macros to use mostly the same tests between the 2 APIs.
• Handle the various exception cases. API key issues return a 403 on AWS with a flat %{"message" => ".."} response, so this falls through to unexpected response. The stream returns empty maps with the error name as the key which need handling in do_process_response. Wondering how to handle these error cases between the stream and non-stream functions.

Was this marged??

[brainlid]

Hi @stevehodgkiss! Where does this stand currently? Do you feel it's ready?

[doomspork]

@stevehodgkiss are there any outstanding changes you need support with to get this across the finish line? Happy to find time if so!

[stevehodgkiss]

Hey @brainlid & @doomspork 👋

Sorry about the delay getting back to this. I've updated the branch with main and tested it with an app that was using a version of this adapter already. Some thoughts on implementation:

• The ChatAnthropic module overrides the default anthropic implementation at a few points (url, headers/signature, errors responses) but the general response shape is the same. There's a refactoring opportunity to move the shared functionality out and have 2 chat model modules instead (ChatBedrock?).
• The test generates 2 versions of the live call tests, one for Anthropics API and another for Bedrocks. Not super keen on this, but it works.

The adapter itself has been working fine, so I'm happy with it functionally.

EDIT: Confirmed this works with the new/upgraded Claude 3.5 Sonnet model on bedrock (anthropic.claude-3-5-sonnet-20241022-v2:0).

[brainlid]

Amazing work @stevehodgkiss! Adding Bedrock support was no easy task.

I think a refactoring opportunity might come when Bedrock support is added to another model.

[brainlid]

Your code looks clean, clear and thoughtful.

❤️💛💙💜