-
Notifications
You must be signed in to change notification settings - Fork 29.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build: run flaky tests in Travis #27158
Conversation
Skipping flaky tests in CI is an anti-pattern that should be avoided, as we do in our ownCI. Failing flaky tests don’t need to be blockers for a green CI result, but they should be run and reported *somehow*.
IMO the travis run should be a minimal sanity test that returns results ASAP. For me test coverage is a non-goal. I'd go as far as advocating for skipping more tests, not less. ATM a typical Travis run takes a minimum of 12.5 minutes: |
Interesting. For me, finishing quickly in Travis is a non-goal. (Finishing quickly for local runs, however, is.) Making Travis as useful as possible is a goal, and that means not skipping tests we don't need to skip. |
I just don't see that as a problem, to be honest. If tests aren't working in a Travis environment, they're probably not going to work for some users in some environments. So I want to know. |
Practically speaking this PR only affects a small number of tests. e.g. for #27164 our CI ran 2615 tests (https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/12353/) while the Travis run ran 2610 tests (https://travis-ci.com/nodejs/node/jobs/191674829#L1063). Five tests isn't going to make a huge difference to the overall running time. |
Landed in 56354d4 |
Skipping flaky tests in CI is an anti-pattern that should be avoided, as we do in our ownCI. Failing flaky tests don’t need to be blockers for a green CI result, but they should be run and reported *somehow*. PR-URL: nodejs#27158 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
* skip compilation by using cache * split testing into two jobs * DRY with YAML anchors PR-URL: nodejs#27182 Refs: nodejs#27158 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rich Trott <rtrott@gmail.com>
Continuation of #27015
Skipping flaky tests in CI is an anti-pattern that should be avoided,
as we do in our ownCI. Failing flaky tests don’t need to be blockers
for a green CI result, but they should be run and reported somehow.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes