Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI: Change to fork of audit to log flaky tests #9518

Merged
merged 19 commits into from
Jan 21, 2021
Merged

Conversation

backspace
Copy link
Contributor

@backspace backspace commented Dec 3, 2020

This produces comments in a PR such as this (fake) one when the test audit runs encounter a flaky test:

Ember Test Audit flaky tests

Ember Test Audit detected these flaky tests on 29bd325:

  • Acceptance | job evaluations: it sometimes fails

This will help in cases such as this one where the test audit comparison shows only that a flaky test exists but not which test(s) it is:

Ember Test Audit comparison

master 94247f3 change
passes 1538 1537 -1
failures 0 0 0
flaky 0 1 +1
duration 7m 46s 439ms 8m 10s 630ms +24s 191ms

Unlike the comparison post, which is “sticky” and therefore gets updated with every run, this posts a comment every time a flaky test is encountered. That might be annoying if there’s a consistently flaky test but seems preferable to me to losing a flaky test name in the edit history of a sticky comment. A possible improvement would be to have a sticky comment with an always-growing list of flaky test names, I can work on that if it seems desirable.

This doesn’t log the names for now.
@vercel
Copy link

vercel bot commented Dec 3, 2020

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

nomad-storybook-and-ui – ./ui

🔍 Inspect: https://vercel.com/hashicorp/nomad-storybook-and-ui/3nnpopspn
✅ Preview: Canceled

[Deployment for 68dbcfe canceled]

@vercel vercel bot temporarily deployed to Preview – nomad December 3, 2020 22:43 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 3, 2020 22:43 Inactive
@backspace backspace marked this pull request as ready for review December 3, 2020 22:52
@vercel vercel bot temporarily deployed to Preview – nomad December 3, 2020 22:55 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 3, 2020 22:55 Inactive
@backspace backspace marked this pull request as draft December 4, 2020 14:44
OF COURSE the action wasn’t running because there were no
changes in ui/ 🥴
@github-actions
Copy link

github-actions bot commented Dec 4, 2020

Ember Asset Size action

As of 99653bb

Files that stayed the same size 🤷‍:

File raw gzip
nomad-ui.js 0 B 0 B
vendor.js 0 B 0 B
nomad-ui.css 0 B 0 B
vendor.css 0 B 0 B

@github-actions
Copy link

github-actions bot commented Dec 4, 2020

Ember Test Audit comparison

master 99653bb change
passes 0 0 0
failures 1 0 -1
flaky 0 1 +1
duration 000ms 892ms +892ms

@vercel vercel bot temporarily deployed to Preview – nomad December 4, 2020 15:54 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 4, 2020 15:54 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 4, 2020 19:27 Inactive
@github-actions
Copy link

github-actions bot commented Dec 4, 2020

Ember Test Audit flaky tests

Ember Test Audit detected these flaky tests on 29bd325:

  • Acceptance | job evaluations: it sometimes fails

@vercel vercel bot temporarily deployed to Preview – nomad December 4, 2020 19:52 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 4, 2020 19:52 Inactive
@github-actions
Copy link

github-actions bot commented Dec 4, 2020

Ember Test Audit flaky tests

Ember Test Audit detected these flaky tests on 99653bb:

  • Acceptance | job evaluations: it sometimes fails

@vercel vercel bot temporarily deployed to Preview – nomad December 14, 2020 21:50 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 14, 2020 21:50 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 16:35 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 16:35 Inactive
@backspace backspace marked this pull request as ready for review January 11, 2021 16:49
Copy link
Contributor

@DingoEatingFuzz DingoEatingFuzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks awesome! I just published v0.4.0 of ember-test-audit after merging your PR so we can go back to using the main package.

@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 20:16 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 20:16 Inactive
@backspace
Copy link
Contributor Author

Updated, thanks for merging that!

Copy link
Contributor

@DingoEatingFuzz DingoEatingFuzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, thought I approved this already. Sorry for the delay.

@backspace backspace merged commit 8847de1 into master Jan 21, 2021
@backspace backspace deleted the f-ui/flaky-audit branch January 21, 2021 14:25
backspace added a commit that referenced this pull request Jan 22, 2021
This will report the names of flaky tests instead of just counting them.
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants