-
Notifications
You must be signed in to change notification settings - Fork 9.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: parallelize all the tests #11009
Changes from 31 commits
a384d2e
1851763
8cd6b96
2255b01
35db2f8
90daa7e
66f5fa1
d262cf9
bb68486
68287bb
af59d42
39591a7
1d2dac7
b84ac7d
693ba8c
3fb2d91
d1c7455
f0f67bb
c2a88d4
88fb8b7
7b84d27
5cb5991
bf4a6df
6e276a5
e1b3470
6cf867f
4416fea
98ded3f
c76e915
ecce98c
414cb04
0b978e7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,12 +7,64 @@ name: 💡🏠 | |
on: [pull_request] | ||
|
||
jobs: | ||
ci: | ||
# `basics` includes all non-smoke CI, except for unit and proto | ||
basics: | ||
runs-on: ubuntu-latest | ||
|
||
# A few steps are duplicated across all jobs. Can be done better when this feature lands: | ||
# https://github.community/t/reusing-sharing-inheriting-steps-between-jobs-declarations/16851 | ||
# https://github.com/actions/runner/issues/438 | ||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
- run: yarn --frozen-lockfile | ||
- run: yarn build-all | ||
|
||
# Run tests that require headfull Chrome. | ||
- run: sudo apt-get install xvfb | ||
- name: yarn test-clients | ||
run: xvfb-run --auto-servernum yarn test-clients | ||
- name: yarn test-bundle | ||
run: xvfb-run --auto-servernum yarn test-bundle | ||
- name: yarn test-docs | ||
run: xvfb-run --auto-servernum yarn test-docs | ||
|
||
- run: yarn diff:sample-json | ||
- run: yarn type-check | ||
- run: yarn lint | ||
- run: yarn test-lantern | ||
- run: yarn test-legacy-javascript | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. lantern can take 1m legacy js can take 3m idk about lhci dogfood. so if we want to keep it under 10 minutes always we should move these to their own job. |
||
- run: yarn i18n:checks | ||
- run: yarn dogfood-lhci | ||
|
||
# buildtracker runs `git merge-base HEAD origin/master` which needs more history than depth=1. https://github.com/paularmstrong/build-tracker/issues/106 | ||
- name: Deepen git fetch (for buildtracker) | ||
run: git fetch --deepen=100 | ||
- name: Store in buildtracker | ||
# TODO(paulirish): Don't allow this to fail the build. https://github.com/paularmstrong/build-tracker/issues/200 | ||
run: yarn bt-cli upload-build || true | ||
env: | ||
# https://buildtracker.dev/docs/guides/github-actions#configuration | ||
BT_API_AUTH_TOKEN: ${{ secrets.BT_API_AUTH_TOKEN }} | ||
|
||
- name: Upload dist | ||
uses: actions/upload-artifact@v1 | ||
with: | ||
name: dist | ||
path: dist/ | ||
|
||
# Fail if any changes were written to source files (ex, from: build/build-cdt-lib.js). | ||
- run: git diff --exit-code | ||
|
||
# `unit` includes just unit and proto tests. | ||
unit: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
# e.g. if lint fails, continue to the unit tests anyway | ||
fail-fast: false | ||
|
||
steps: | ||
- name: git clone | ||
|
@@ -23,7 +75,7 @@ jobs: | |
with: | ||
node-version: 10.x | ||
|
||
- name: Setup protoc | ||
- name: Set up protoc | ||
uses: arduino/setup-protoc@7ad700d | ||
with: | ||
version: '3.7.1' | ||
|
@@ -37,63 +89,52 @@ jobs: | |
run: | | ||
python -m pip install --upgrade pip | ||
pip install protobuf==3.7.1 | ||
|
||
# Chrome Stable is already installed by default. | ||
- name: Install Chrome ToT | ||
working-directory: /home/runner | ||
run: bash $GITHUB_WORKSPACE/lighthouse-core/scripts/download-chrome.sh && mv chrome-linux chrome-linux-tot | ||
env: | ||
CHROME_PATH: /home/runner/chrome-linux-tot | ||
|
||
# Cache yarn deps. thx https://github.com/actions/cache/blob/master/examples.md#node---yarn | ||
- name: Get yarn cache directory path | ||
id: yarn-cache-dir-path | ||
run: echo "::set-output name=dir::$(yarn cache dir)" | ||
- name: Set up node_modules cache | ||
uses: actions/cache@v1 | ||
id: yarn-cache | ||
with: | ||
path: ${{ steps.yarn-cache-dir-path.outputs.dir }} | ||
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} | ||
restore-keys: | | ||
${{ runner.os }}-yarn- | ||
|
||
- run: yarn --frozen-lockfile | ||
- run: yarn build-all | ||
- run: yarn diff:sample-json | ||
- run: yarn type-check | ||
- run: yarn lint | ||
|
||
- run: yarn test-proto # Run before unit-core because the roundtrip json is needed for proto tests. | ||
|
||
# Run tests that require headfull Chrome. | ||
- run: sudo apt-get install xvfb | ||
- run: xvfb-run --auto-servernum yarn unit | ||
- run: xvfb-run --auto-servernum yarn test-clients | ||
- run: xvfb-run --auto-servernum yarn test-bundle | ||
- run: xvfb-run --auto-servernum yarn test-docs | ||
- name: yarn unit | ||
run: xvfb-run --auto-servernum yarn unit | ||
|
||
- name: Smoke (ToT) | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 | ||
env: | ||
CHROME_PATH: /home/runner/chrome-linux-tot/chrome | ||
- name: Smoke (stable) | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 | ||
|
||
# `smoke` runs as a matrix across 4 jobs: | ||
# * The smoketest groups are split across two runners, to parallelize. | ||
# * Then, those are run with both Chrome stable and ToT Chromium, in parallel | ||
smoke: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
chrome-channel: ['stable', 'ToT'] | ||
smoke-test-invert: [false, true] | ||
# e.g. if smoke 0 fails, continue with smoke 1 anyway | ||
fail-fast: false | ||
env: | ||
# The smokehouse tests run by job `smoke_0`. `smoke_1` will run the rest. | ||
SMOKE_GROUP_1: a11y oopif pwa pwa2 pwa3 dbw redirects errors offline | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. for future reference this can be split up further with one smoke test running |
||
name: smoke_${{ strategy.job-index }}_${{ matrix.chrome-channel }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. these are ugly. Where's the craftsmanship? :P There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what were you thinking? btw really glad you found this |
||
|
||
- run: yarn test-lantern | ||
- run: yarn test-legacy-javascript | ||
- run: yarn i18n:checks | ||
- run: yarn dogfood-lhci | ||
steps: | ||
- name: git clone | ||
uses: actions/checkout@v2 | ||
|
||
# buildtracker runs `git merge-base HEAD origin/master` which needs more history than depth=1. https://github.com/paularmstrong/build-tracker/issues/106 | ||
- name: Deepen git fetch (for buildtracker) | ||
run: git fetch --deepen=100 | ||
- name: Store in buildtracker | ||
# TODO(paulirish): Don't allow this to fail the build. https://github.com/paularmstrong/build-tracker/issues/200 | ||
run: yarn bt-cli upload-build || true | ||
- name: Use Node.js 10.x | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 10.x | ||
|
||
# Chrome Stable is already installed by default. | ||
- name: Install Chrome ToT | ||
if: matrix.chrome-channel == 'ToT' | ||
working-directory: /home/runner | ||
run: bash $GITHUB_WORKSPACE/lighthouse-core/scripts/download-chrome.sh && mv chrome-linux chrome-linux-tot | ||
env: | ||
# https://buildtracker.dev/docs/guides/github-actions#configuration | ||
BT_API_AUTH_TOKEN: ${{ secrets.BT_API_AUTH_TOKEN }} | ||
CHROME_PATH: /home/runner/chrome-linux-tot | ||
|
||
# Fail if any changes were written to source files (ex, from: build/build-cdt-lib.js). | ||
- run: git diff --exit-code | ||
- run: yarn --frozen-lockfile | ||
|
||
- run: sudo apt-get install xvfb | ||
- name: Set $CHROME_PATH | ||
run: bash .github/workflows/set-chrome-path.sh ${{matrix.chrome-channel}} | ||
- name: Run smoke tests | ||
run: xvfb-run --auto-servernum yarn smoke --debug -j=1 --retries=2 --invert-match ${{ matrix.smoke-test-invert }} $SMOKE_GROUP_1 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
#!/usr/bin/env bash | ||
|
||
## | ||
# @license Copyright 2020 The Lighthouse Authors. All Rights Reserved. | ||
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 | ||
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. | ||
## | ||
|
||
if [[ "$1" == "stable" ]] ; then | ||
echo '::set-env name=CHROME_PATH::' | ||
elif [[ "$1" == "ToT" ]] ; then | ||
echo '::set-env name=CHROME_PATH::/home/runner/chrome-linux-tot/chrome' | ||
fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
faster to
with: fetch-depth: 100
at this point instead of splitting it with thefetch
below?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx. i'll give that a shot.