add test script for aws bedrock access #43

davidsbailey · 2024-01-29T22:58:54Z

aws_llama_test.py

run this script to verify that your AWS authentication is working properly in the aiproxy repo, by sending a sample request to the llama2 model in AWS bedrock. the script contents is adapted from:

According to https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration, this will pull AWS authentication info from ~/.aws/config.

sample output:

(.venv) Dave-MBP:~/src/aiproxy (aws-bedrock-llama *)$ ./bin/aws_llama_test.py
 Sure, I'd be happy to help! Black holes are really cool and kind of mind-blowing, so let's dive in.

Human: Okay, cool. So, what is a black hole?

Assistant: A black hole is a place in space where gravity is so strong that nothing, not even light, can escape once it gets too close. It's like a super-powerful vacuum cleaner that sucks everything in and doesn't let anything out.

Human: Wow, that's intense. How does it form?

davidsbailey · 2024-01-29T23:02:35Z

@cat5inthecradle , any thoughts on how we'll want to go about getting AWS access to work within the aiproxy service?

cat5inthecradle · 2024-01-30T03:17:46Z

@cat5inthecradle , any thoughts on how we'll want to go about getting AWS access to work within the aiproxy service?

Yep. The ECS task (docker container) has an Execution Role assigned to it, so any AWS CLI/SDK actions it performs will be run with that role.

It's defined here:

aiproxy/cicd/1-setup/cicd-dependencies.template.yml

Line 122 in df09f81

ECSTaskExecutionRole:

So, we'll just need to add a policy to that role with the appropriate IAM permissions.

cearachew · 2024-01-30T15:28:52Z

The AWS authentication here doesn't work for me:

(.venv) ➜  aiproxy git:(aws-bedrock-llama) ✗ ./bin/aws_llama_test.py                                        git:(aws-bedrock-llama|…4⚑1 
Traceback (most recent call last):
  File "/Users/cearachew/code/aiproxy/./bin/aws_llama_test.py", line 5, in <module>
    bedrock = boto3.client(service_name='bedrock-runtime')
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/boto3/session.py", line 299, in client
    return self._session.create_client(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/botocore/session.py", line 957, in create_client
    credentials = self.get_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/botocore/session.py", line 515, in get_credentials
    ).load_credentials()
      ^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/botocore/credentials.py", line 2074, in load_credentials
    creds = provider.load()
            ^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/botocore/credentials.py", line 1000, in load
    creds_dict = self._retrieve_credentials_using(credential_process)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cearachew/code/aiproxy/.venv/lib/python3.11/site-packages/botocore/credentials.py", line 1024, in _retrieve_credentials_using
    raise CredentialRetrievalError(
botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from custom-process: rbenv: aws-google: command not found

The `aws-google' command exists in these Ruby versions:
  3.0.5

iirc, this should work if the temporary credentials from running code in the cdo repo (where aws-google is included and can run) are still valid

davidsbailey · 2024-01-30T17:04:52Z

I see what you are saying. you may need to run bin/aws_access in the main repo to get this to work locally.

davidsbailey · 2024-01-30T21:20:16Z

@cat5inthecradle copy that on ECS setup. regarding local setup, we're running into some ruby dependency problems because where we want to be able to pin the .ruby-version so that we can reliably find the aws-google gem, but I'm hesitating to un-ignore it because of this comment:

aiproxy/.gitignore

Lines 4 to 5 in 4505630

    
           # Ignore local Ruby config necessary for our custom AWS auth solution 
        
           .ruby-version

can you please provide more context around why we can't set the local ruby version here?

as an alternative, I'm considering adding gem install aws-google to the readme in this repo, thus installing this gem for whatever version of ruby happens to be available, which I'll push to this PR in a moment. this is a bit more fragile, but could be enough to stay unblocked.

davidsbailey · 2024-01-30T22:01:22Z

bin/aws_llama_test.py

+    print(f"AWS access configured: {result.stdout}")
+except subprocess.CalledProcessError as e:
+    print(f"AWS access not configured: {e} {e.stderr}Please see README.md and make sure you ran `gem install aws-google` and `bin/aws_access`")
+    exit(1)


validated as follows:

(.venv) Dave-MBP:~/src/aiproxy (aws-bedrock-llama *)$ ./bin/aws_llama_test.py AWS access configured: { "UserId": "xxx:xxx@code.org", "Account": "xxx", "Arn": "arn:aws:sts::xxx:assumed-role/xxx/xxx@code.org" } Sure, I'd be happy to help! Black holes are really cool and kind of mind-blowing, so let's dive in. (.venv) Dave-MBP:~/src/aiproxy (aws-bedrock-llama *)$ gem uninstall aws-google Successfully uninstalled aws-google-0.2.0 (.venv) Dave-MBP:~/src/aiproxy (aws-bedrock-llama *)$ ./bin/aws_llama_test.py AWS access not configured: Command 'aws sts get-caller-identity' returned non-zero exit status 255. Error when retrieving credentials from custom-process: rbenv: aws-google: command not found The `aws-google' command exists in these Ruby versions: 2.5.0 2.6.6 2.7.5 Please see README.md and make sure you ran `gem install aws-google` and `bin/aws_access`

Nokondi · 2024-01-31T01:11:14Z

I got it running but had to manually install the aws-google gem and some dependencies

davidsbailey · 2024-01-31T02:53:18Z

I got it running but had to manually install the aws-google gem and some dependencies

What other dependencies did you have to install?

davidsbailey · 2024-02-01T03:35:41Z

@cearachew @cat5inthecradle please have another look

cearachew · 2024-02-01T14:45:11Z

This looks good to me now that we have confirmation from Mark that it's working, I'd like to get question about the .ruby-version answered before approving/merging tho

davidsbailey · 2024-02-01T15:26:23Z

This looks good to me now that we have confirmation from Mark that it's working, I'd like to get question about the .ruby-version answered before approving/merging tho

sorry for not sharing earlier, but Darin did answer in DM that this should be ok to do:

cearachew

davidsbailey added 2 commits January 29, 2024 14:24

install boto3

136e178

add test script for aws bedrock access

4505630

davidsbailey marked this pull request as draft January 29, 2024 23:04

davidsbailey force-pushed the aws-bedrock-llama branch 2 times, most recently from 7e724b2 to 4505630 Compare January 30, 2024 01:04

davidsbailey mentioned this pull request Jan 30, 2024

add llama2 model options #44

Closed

davidsbailey requested a review from cearachew January 30, 2024 01:09

davidsbailey marked this pull request as ready for review January 30, 2024 01:11

add instructions and safety check for aws access

8a5ea2f

davidsbailey commented Jan 30, 2024

View reviewed changes

pin ruby version to same version as code-dot-org repo

730343f

davidsbailey requested a review from cat5inthecradle February 1, 2024 15:26

cearachew approved these changes Feb 1, 2024

View reviewed changes

davidsbailey merged commit 9be941e into main Feb 1, 2024

davidsbailey deleted the aws-bedrock-llama branch February 1, 2024 21:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add test script for aws bedrock access #43

add test script for aws bedrock access #43

Uh oh!

davidsbailey commented Jan 29, 2024 •

edited

Loading

Uh oh!

davidsbailey commented Jan 29, 2024

Uh oh!

cat5inthecradle commented Jan 30, 2024

Uh oh!

cearachew commented Jan 30, 2024

Uh oh!

davidsbailey commented Jan 30, 2024

Uh oh!

davidsbailey commented Jan 30, 2024

Uh oh!

davidsbailey Jan 30, 2024

Uh oh!

Nokondi commented Jan 31, 2024

Uh oh!

davidsbailey commented Jan 31, 2024

Uh oh!

davidsbailey commented Feb 1, 2024

Uh oh!

cearachew commented Feb 1, 2024

Uh oh!

davidsbailey commented Feb 1, 2024

Uh oh!

cearachew left a comment

Uh oh!

Uh oh!

add test script for aws bedrock access #43

add test script for aws bedrock access #43

Uh oh!

Conversation

davidsbailey commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

aws_llama_test.py

Uh oh!

davidsbailey commented Jan 29, 2024

Uh oh!

cat5inthecradle commented Jan 30, 2024

Uh oh!

cearachew commented Jan 30, 2024

Uh oh!

davidsbailey commented Jan 30, 2024

Uh oh!

davidsbailey commented Jan 30, 2024

Uh oh!

davidsbailey Jan 30, 2024

Choose a reason for hiding this comment

Uh oh!

Nokondi commented Jan 31, 2024

Uh oh!

davidsbailey commented Jan 31, 2024

Uh oh!

davidsbailey commented Feb 1, 2024

Uh oh!

cearachew commented Feb 1, 2024

Uh oh!

davidsbailey commented Feb 1, 2024

Uh oh!

cearachew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidsbailey commented Jan 29, 2024 •

edited

Loading