-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Integrate automated smoke test into release workflow #153
test: Integrate automated smoke test into release workflow #153
Conversation
.github/scripts/upload_smoke_test.sh
Outdated
@@ -0,0 +1,2 @@ | |||
bucket=$1 | |||
aws s3api put-object --bucket $bucket --key "smoke.html" --body processed_smoke.html --content-type "text/html" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: Would it be difficult to version the smoke test file? E.g., smoke-1.5.1.html
? This way we could (1) retain older versions for testing, and (2) it would be easy to verify which versions have been tested.
This isn't a blocker, but something to consider if it's easy to do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to do so, I think the changes need to happen on (1) CDK to edit the bucket with versioning characteristics (2) upload_smoke_test.sh
and cd.yaml
.
I think it's better to have the fix incorporated with this, since otherwise we will have to keep the enhancement in our backlog (which might get overlooked).
I will add the changes with the next revision.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clarification: Bucket versioning is not needed - we expect the bucket to hold different versions of the smoke test, with the naming convention of *.cloudfront.net/smoke-{version}.html
Verified testing in gamma
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm pending @limhjgrace question
21866b8
to
a56f22d
Compare
New Tests:
Verified:
|
Details
Currently as part of the release process, contributors in AWS RUM manually perform smoke tests to gain assurance on the newly-releasing web client version's behavior. This process has a couple of issues:
To mitigate this, this PR integrates automated smoke test as part of the release workflow.
Changes
smoke.html
into repository, with all sensitive parameters anonymized and stored in GitHub secrets.Testing
Setup
This is only applicable to AWS RUM members.
Notes
When the test is run with 4 workers, all the ingested events were retrieved within a maximum of 2 minutes. However, when the tests are run with 6 workers, around 1-2 cases time out (5m) and the overall time to retrieve events is increased to 2-3 minutes.
AWS RUM team should investigate into this issue.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.