-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UI: Change to fork of audit to log flaky tests #9518
Conversation
This doesn’t log the names for now.
This pull request is being automatically deployed with Vercel (learn more). nomad-storybook-and-ui – ./ui🔍 Inspect: https://vercel.com/hashicorp/nomad-storybook-and-ui/3nnpopspn [Deployment for 68dbcfe canceled] |
OF COURSE the action wasn’t running because there were no changes in ui/ 🥴
Ember Asset Size actionAs of 99653bb Files that stayed the same size 🤷:
|
Ember Test Audit comparison
|
Ember Test Audit flaky testsEmber Test Audit detected these flaky tests on 29bd325:
|
Ember Test Audit flaky testsEmber Test Audit detected these flaky tests on 99653bb:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome! I just published v0.4.0 of ember-test-audit
after merging your PR so we can go back to using the main package.
Updated, thanks for merging that! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops, thought I approved this already. Sorry for the delay.
This will report the names of flaky tests instead of just counting them.
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
This produces comments in a PR such as this (fake) one when the test audit runs encounter a flaky test:
This will help in cases such as this one where the test audit comparison shows only that a flaky test exists but not which test(s) it is:
Unlike the comparison post, which is “sticky” and therefore gets updated with every run, this posts a comment every time a flaky test is encountered. That might be annoying if there’s a consistently flaky test but seems preferable to me to losing a flaky test name in the edit history of a sticky comment. A possible improvement would be to have a sticky comment with an always-growing list of flaky test names, I can work on that if it seems desirable.