Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add error message test on S3 access decline #5825

Merged
merged 4 commits into from
May 22, 2023
Merged

Add error message test on S3 access decline #5825

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4abef32
Add test on access decline
kthui May 19, 2023
ab11da0
Fix typo
kthui May 19, 2023
0a83103
Add MinIO S3 access decline test
kthui May 20, 2023
9cefd19
Make sure bucket exists during access decline test
kthui May 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion qa/L0_storage_S3/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ aws configure set default.region $AWS_DEFAULT_REGION && \
# Copy models into S3 bucket
aws s3 cp tmp_cred_test_models/ "${BUCKET_URL_SLASH}" --recursive --include "*"

SERVER_ARGS="--model-repository=$ROOT_REPO --exit-timeout-secs=120"
SERVER_ARGS="--model-repository=$BUCKET_URL --exit-timeout-secs=120"

run_server
if [ "$SERVER_PID" == "0" ]; then
Expand Down Expand Up @@ -386,6 +386,36 @@ set -e
kill $SERVER_PID
wait $SERVER_PID

# Test access decline
export AWS_SECRET_ACCESS_KEY="[Invalid]" && export AWS_SESSION_TOKEN=""
SERVER_ARGS="--model-repository=$BUCKET_URL --exit-timeout-secs=120"
run_server
if [ "$SERVER_PID" != "0" ]; then
echo -e "\n***\n*** Unexpected server start $SERVER\n***"
cat $SERVER_LOG
kill $SERVER_PID
wait $SERVER_PID
RET=1
else
# AWS S3 does not appear to reply on access decline, but other implementations
# might provide extra messages, so make sure Triton will print the messages.
Comment on lines +400 to +401
Copy link
Collaborator

@rmccorm4 rmccorm4 May 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add similar tests to https://github.com/triton-inference-server/server/blob/main/qa/L0_storage_S3_local/test.sh to see how an alternative/on-prem S3 API implementation (minio) replies? I wonder if it is different.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Add MinIO S3 access decline test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the response is the same, "No response body."

EXPECTED_MSG="Unable to create S3 filesystem client. Check account credentials. Exception: '' Message: 'No response body.'"
if ! grep "$EXPECTED_MSG" $SERVER_LOG; then
echo -e "\n***\n*** Expected error message not found\n***"
cat $SERVER_LOG
RET=1
fi
fi

# Restore S3 credentials
rm ~/.aws/credentials && rm ~/.aws/config
export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION_BACKUP
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_BACKUP
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_BACKUP
aws configure set default.region $AWS_DEFAULT_REGION && \
aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID && \
aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY

# Clean up bucket contents and delete bucket
aws s3 rm "${BUCKET_URL_SLASH}" --recursive --include "*"
aws s3 rb "${BUCKET_URL}"
Expand Down
22 changes: 22 additions & 0 deletions qa/L0_storage_S3_local/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,28 @@ set -e
kill $SERVER_PID
wait $SERVER_PID

# Test access decline
export AWS_SECRET_ACCESS_KEY="[Invalid]"
SERVER_ARGS="--model-repository=s3://localhost:4572/${BUCKET_NAME}1 --exit-timeout-secs=120"
SERVER_LOG="./inference_server.access_decline.log"
run_server
if [ "$SERVER_PID" != "0" ]; then
echo -e "\n***\n*** Unexpected server start $SERVER\n***"
cat $SERVER_LOG
kill $SERVER_PID
wait $SERVER_PID
RET=1
else
# MinIO does not appear to reply on access decline, but other implementations
# might provide extra messages, so make sure Triton will print the messages.
EXPECTED_MSG="Unable to create S3 filesystem client. Check account credentials. Exception: '' Message: 'No response body.'"
if ! grep "$EXPECTED_MSG" $SERVER_LOG; then
echo -e "\n***\n*** Expected error message not found\n***"
cat $SERVER_LOG
RET=1
fi
fi

# Destroy buckets
for BUCKET_SUFFIX in 1 2; do
awslocal $ENDPOINT_FLAG s3 rm s3://$BUCKET_NAME$BUCKET_SUFFIX --recursive --include "*" && \
Expand Down