-
Notifications
You must be signed in to change notification settings - Fork 9.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: parallelize all the tests #11009
Changes from all commits
a384d2e
1851763
8cd6b96
2255b01
35db2f8
90daa7e
66f5fa1
d262cf9
bb68486
68287bb
af59d42
39591a7
1d2dac7
b84ac7d
693ba8c
3fb2d91
d1c7455
f0f67bb
c2a88d4
88fb8b7
7b84d27
5cb5991
bf4a6df
6e276a5
e1b3470
6cf867f
4416fea
98ded3f
c76e915
ecce98c
414cb04
0b978e7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,13 +7,13 @@ name: 💡🏠 | |
on: [pull_request] | ||
|
||
jobs: | ||
ci: | ||
|
||
# `basics` includes all non-smoke CI, except for unit and proto | ||
basics: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
# e.g. if lint fails, continue to the unit tests anyway | ||
fail-fast: false | ||
|
||
# A few steps are duplicated across all jobs. Can be done better when this feature lands: | ||
# https://github.community/t/reusing-sharing-inheriting-steps-between-jobs-declarations/16851 | ||
# https://github.com/actions/runner/issues/438 | ||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
@@ -23,63 +23,21 @@ jobs: | |
with: | ||
node-version: 10.x | ||
|
||
- name: Setup protoc | ||
uses: arduino/setup-protoc@7ad700d | ||
with: | ||
version: '3.7.1' | ||
repo-token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v1 | ||
with: | ||
python-version: 2.7 | ||
- name: Install Python dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install protobuf==3.7.1 | ||
|
||
# Chrome Stable is already installed by default. | ||
- name: Install Chrome ToT | ||
working-directory: /home/runner | ||
run: bash $GITHUB_WORKSPACE/lighthouse-core/scripts/download-chrome.sh && mv chrome-linux chrome-linux-tot | ||
env: | ||
CHROME_PATH: /home/runner/chrome-linux-tot | ||
|
||
# Cache yarn deps. thx https://github.com/actions/cache/blob/master/examples.md#node---yarn | ||
- name: Get yarn cache directory path | ||
id: yarn-cache-dir-path | ||
run: echo "::set-output name=dir::$(yarn cache dir)" | ||
- name: Set up node_modules cache | ||
uses: actions/cache@v1 | ||
id: yarn-cache | ||
with: | ||
path: ${{ steps.yarn-cache-dir-path.outputs.dir }} | ||
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} | ||
restore-keys: | | ||
${{ runner.os }}-yarn- | ||
|
||
- run: yarn --frozen-lockfile | ||
- run: yarn build-all | ||
- run: yarn diff:sample-json | ||
- run: yarn type-check | ||
- run: yarn lint | ||
- run: yarn test-proto # Run before unit-core because the roundtrip json is needed for proto tests. | ||
|
||
# Run tests that require headfull Chrome. | ||
- run: sudo apt-get install xvfb | ||
- run: xvfb-run --auto-servernum yarn unit | ||
- run: xvfb-run --auto-servernum yarn test-clients | ||
- run: xvfb-run --auto-servernum yarn test-bundle | ||
- run: xvfb-run --auto-servernum yarn test-docs | ||
|
||
- name: Smoke (ToT) | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 | ||
env: | ||
CHROME_PATH: /home/runner/chrome-linux-tot/chrome | ||
- name: Smoke (stable) | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 | ||
|
||
- name: yarn test-clients | ||
run: xvfb-run --auto-servernum yarn test-clients | ||
- name: yarn test-bundle | ||
run: xvfb-run --auto-servernum yarn test-bundle | ||
- name: yarn test-docs | ||
run: xvfb-run --auto-servernum yarn test-docs | ||
|
||
- run: yarn diff:sample-json | ||
- run: yarn type-check | ||
- run: yarn lint | ||
- run: yarn test-lantern | ||
- run: yarn test-legacy-javascript | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. lantern can take 1m legacy js can take 3m idk about lhci dogfood. so if we want to keep it under 10 minutes always we should move these to their own job. |
||
- run: yarn i18n:checks | ||
|
@@ -95,5 +53,89 @@ jobs: | |
# https://buildtracker.dev/docs/guides/github-actions#configuration | ||
BT_API_AUTH_TOKEN: ${{ secrets.BT_API_AUTH_TOKEN }} | ||
|
||
- name: Upload dist | ||
uses: actions/upload-artifact@v1 | ||
with: | ||
name: dist | ||
path: dist/ | ||
|
||
# Fail if any changes were written to source files (ex, from: build/build-cdt-lib.js). | ||
- run: git diff --exit-code | ||
|
||
# `unit` includes just unit and proto tests. | ||
unit: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
- name: Set up protoc | ||
uses: arduino/setup-protoc@7ad700d | ||
with: | ||
version: '3.7.1' | ||
repo-token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v1 | ||
with: | ||
python-version: 2.7 | ||
- name: Install Python dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install protobuf==3.7.1 | ||
|
||
- run: yarn --frozen-lockfile | ||
|
||
- run: yarn test-proto # Run before unit-core because the roundtrip json is needed for proto tests. | ||
|
||
- run: sudo apt-get install xvfb | ||
- name: yarn unit | ||
run: xvfb-run --auto-servernum yarn unit | ||
|
||
# `smoke` runs as a matrix across 4 jobs: | ||
# * The smoketest groups are split across two runners, to parallelize. | ||
# * Then, those are run with both Chrome stable and ToT Chromium, in parallel | ||
smoke: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
chrome-channel: ['stable', 'ToT'] | ||
smoke-test-invert: [false, true] | ||
# e.g. if smoke 0 fails, continue with smoke 1 anyway | ||
fail-fast: false | ||
env: | ||
# The smokehouse tests run by job `smoke_0`. `smoke_1` will run the rest. | ||
SMOKE_GROUP_1: a11y oopif pwa pwa2 pwa3 dbw redirects errors offline | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. for future reference this can be split up further with one smoke test running |
||
name: smoke_${{ strategy.job-index }}_${{ matrix.chrome-channel }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. these are ugly. Where's the craftsmanship? :P There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what were you thinking? btw really glad you found this |
||
|
||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
- name: Define ToT chrome path | ||
if: matrix.chrome-channel == 'ToT' | ||
run: echo "::set-env name=CHROME_PATH::/home/runner/chrome-linux-tot/chrome" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think set-env only applies to the next step. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yeah the wording is stupid https://help.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-environment-variable
"Next", you say...
so yeah. it's all subsequent. also i hate how this doc says "actions" but should say "steps". tested it earlier: https://github.com/GoogleChrome/lighthouse/runs/805258701?check_suite_focus=true I read it two steps after i set it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ah good catch. cc @github lol |
||
|
||
# Chrome Stable is already installed by default. | ||
- name: Install Chrome ToT | ||
if: matrix.chrome-channel == 'ToT' | ||
working-directory: /home/runner | ||
run: bash $GITHUB_WORKSPACE/lighthouse-core/scripts/download-chrome.sh && mv chrome-linux chrome-linux-tot | ||
|
||
- run: yarn --frozen-lockfile | ||
|
||
- run: sudo apt-get install xvfb | ||
|
||
- name: Run smoke tests | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 --invert-match ${{ matrix.smoke-test-invert }} $SMOKE_GROUP_1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
faster to
with: fetch-depth: 100
at this point instead of splitting it with thefetch
below?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx. i'll give that a shot.