-
Notifications
You must be signed in to change notification settings - Fork 9.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: run smoke tests in parallel jobs #10993
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,12 +7,26 @@ name: π‘π | |
on: [pull_request] | ||
|
||
jobs: | ||
ci: | ||
basics: | ||
# basic checks that depend only on the cloned and yarned repo. | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
- run: yarn --frozen-lockfile | ||
- run: yarn type-check | ||
- run: yarn lint | ||
- run: yarn i18n:checks | ||
|
||
ci: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
# e.g. if lint fails, continue to the unit tests anyway | ||
fail-fast: false | ||
|
||
steps: | ||
- name: git clone | ||
|
@@ -23,7 +37,7 @@ jobs: | |
with: | ||
node-version: 10.x | ||
|
||
- name: Setup protoc | ||
- name: Set up protoc | ||
uses: arduino/setup-protoc@7ad700d | ||
with: | ||
version: '3.7.1' | ||
|
@@ -38,39 +52,35 @@ jobs: | |
python -m pip install --upgrade pip | ||
pip install protobuf==3.7.1 | ||
|
||
# Cache yarn deps. thx https://github.com/actions/cache/blob/master/examples.md#node---yarn | ||
- name: Get yarn cache directory path | ||
id: yarn-cache-dir-path | ||
run: echo "::set-output name=dir::$(yarn cache dir)" | ||
- name: Set up node_modules cache | ||
uses: actions/cache@v1 | ||
id: yarn-cache | ||
with: | ||
path: ${{ steps.yarn-cache-dir-path.outputs.dir }} | ||
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} | ||
restore-keys: | | ||
${{ runner.os }}-yarn- | ||
|
||
- run: yarn --frozen-lockfile | ||
- run: yarn build-all | ||
- run: yarn diff:sample-json | ||
- run: yarn type-check | ||
- run: yarn lint | ||
- run: yarn test-proto # Run before unit-core because the roundtrip json is needed for proto tests. | ||
|
||
# Run tests that require headfull Chrome. | ||
- run: sudo apt-get install xvfb | ||
- run: xvfb-run --auto-servernum yarn unit | ||
- run: xvfb-run --auto-servernum yarn test-clients | ||
- run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 | ||
- run: xvfb-run --auto-servernum yarn test-bundle | ||
- run: xvfb-run --auto-servernum yarn test-docs | ||
- name: yarn unit | ||
run: xvfb-run --auto-servernum yarn unit | ||
- name: yarn test-clients | ||
run: xvfb-run --auto-servernum yarn test-clients | ||
- name: yarn test-bundle | ||
run: xvfb-run --auto-servernum yarn test-bundle | ||
- name: yarn test-docs | ||
run: xvfb-run --auto-servernum yarn test-docs | ||
|
||
- run: yarn test-lantern | ||
- run: yarn test-legacy-javascript | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some of these tests can take a long time. That's why I split up the "special" tests (legacy js, lantern, etc.) from the core unit test and the build jobβwith this PR all of these are still in the same job. |
||
- run: yarn i18n:checks | ||
- run: yarn dogfood-lhci | ||
|
||
# Fail if any changes were written to source files (ex, from: build/build-cdt-lib.js). | ||
- run: git diff --exit-code | ||
|
||
- name: Upload dist/ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. kept from #10988 |
||
uses: actions/upload-artifact@v2 | ||
with: | ||
name: dist | ||
path: dist/ | ||
|
||
# buildtracker runs `git merge-base HEAD origin/master` which needs more history than depth=1. https://github.com/paularmstrong/build-tracker/issues/106 | ||
- name: Deepen git fetch (for buildtracker) | ||
run: git fetch --deepen=100 | ||
|
@@ -81,5 +91,27 @@ jobs: | |
# https://buildtracker.dev/docs/guides/github-actions#configuration | ||
BT_API_AUTH_TOKEN: ${{ secrets.BT_API_AUTH_TOKEN }} | ||
|
||
# Fail if any changes were written to source files (ex, from: build/build-cdt-lib.js). | ||
- run: git diff --exit-code | ||
smoke: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
smokeTestInvert: [false, true] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yaml keys are normally skewer-case. |
||
# e.g. if smoke 0 fails, continue with smoke 1 anyway | ||
fail-fast: false | ||
env: | ||
# The smokehouse tests run by job smoke 0. smoke-1 will run the rest. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. whatcha thinking with ToTChrome in the mix too? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
matrix:
smokeTestInvert: [false, true]
chrome: [Stable, ToT]
name: smoke ${{ strategy.job-index }} (Chrome ${{ matrix.chrome }}) and then conditionally set There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I broke up the ToT and parallel job PRs for a reason. Things are getting complex :) |
||
SMOKE_GROUP_1: a11y oopif pwa pwa2 pwa3 dbw redirects errors offline | ||
name: smoke ${{ strategy.job-index }} | ||
|
||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
- run: yarn --frozen-lockfile | ||
- name: Run smoke tests | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 --invert-match ${{ matrix.smokeTestInvert }} $SMOKE_GROUP_1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looking at the runs for https://github.com/GoogleChrome/lighthouse/pull/10988/checks?check_run_id=783384854 and running a bunch of them myself, it seems like caching only saves like 5 seconds out of ~30. Doesn't seem worth it.