Skip to content

Commit d73e0fd

Browse files
authored
Document the process and get things ready
1 parent ba5f1a3 commit d73e0fd

File tree

8 files changed

+283
-2
lines changed

8 files changed

+283
-2
lines changed

.cirrus.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
task:
2+
name: tests-linux
3+
use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
4+
script: scripts/verify_tests_on_master.sh
5+
container:
6+
image: gcc:latest
7+
8+
task:
9+
name: tests-windows
10+
use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
11+
script: scripts\verify_tests_on_master.bat
12+
windows_container:
13+
image: cirrusci/windowsservercore:2019
14+
os_version: 2019
15+
16+
task:
17+
name: tests-macos
18+
use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
19+
script: scripts/verify_tests_on_master.sh
20+
osx_instance:
21+
image: mojave-base

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
flutter

README.md

Lines changed: 110 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,110 @@
1-
# tests
2-
Contributed tests for Flutter
1+
# Flutter Tests
2+
3+
This repository contains references to tests that are run with every
4+
commit to Flutter to verify that no breaking changes have been
5+
introduced. The tests referenced by this repository are typically
6+
maintained by people outside of the Flutter team, as part of the
7+
development of their applications. They are intended to give the
8+
Flutter team visibility into how their changes affect real-world
9+
developers using Flutter.
10+
11+
## Adding more tests
12+
13+
You are welcome to add a new batch of tests. To do so, copy the file
14+
`registry/template.test` to create a new file in the `registry/`
15+
directory. Fill in the fields and delete all the comments. Then,
16+
submit a PR with this new file.
17+
18+
Tests must fulfill the following criteria to be added:
19+
20+
* All the code must be available publicly on GitHub under a license
21+
compatible with this effort.
22+
23+
* Tests must be hermetic. For example, a test should not involve
24+
network activity, spawn processes, or access the local file system
25+
except to access files that are packaged with the test.
26+
27+
* Tests must be resilient to being run concurrently with other tests,
28+
including concurrent runs of themselves.
29+
30+
* Tests must be reliable. A test must not claim to pass if it is
31+
failing. Running a test multiple times in a row must always have the
32+
same result.
33+
34+
* Tests must have no output when they are passing.
35+
36+
* Tests must be as fast as possible given the hardware. For example,
37+
tests must not use real timers or depend on the wall clock.
38+
39+
* The time taken by tests must be proportional to their value. A few
40+
thousands tests are expected to run within a few minutes. An upper
41+
limit of about five minutes will be applied to each contributed test
42+
suite (not including the time to download the tests), but it is
43+
expected that most suites will complete in seconds.
44+
45+
* The tests must be compatible with any tools for automatically
46+
updating Flutter code (e.g. they cannot rely on custom code
47+
generation unless such code generation can hook into the automatic
48+
update mechanism).
49+
50+
* The tests must represent good practices as described in Flutter's
51+
documentation. For example, using an object after calling its
52+
`dispose` method violates the contract described by that method.
53+
Accessing the fields of a private `State` subclass from another
54+
package by casting it to dynamic is similarly sketchy and would not
55+
be supported behaviour.
56+
57+
* The tests must pass at the time they are contributed.
58+
59+
60+
## Running the tests locally
61+
62+
To run these tests locally, check out this directory in a directory
63+
parallel to your `flutter` repository checkout, then, from this
64+
directory, run:
65+
66+
```
67+
../flutter/bin/cache/dart-sdk/bin/dart ../flutter/dev/customer_testing/run_tests.dart --skip-template --verbose registry/*.test
68+
```
69+
70+
71+
## If a test is broken
72+
73+
The point of these tests is to make sure we don't break existing code,
74+
while still being able to make improvements to Flutter APIs.
75+
76+
If you find that a PR you have created in flutter/flutter causes one
77+
these tests to fail, you have the following options:
78+
79+
1. Change your PR so that the test no longer fails. This is the
80+
preferred option, so long as the result is one we can be proud of.
81+
Is the resulting API something that you would plausibly come up
82+
with without the backwards-compatibility constraint? That's good.
83+
Is the resulting API something that, as soon as you see it, you
84+
think "why?" or "that's weird"? That's bad. Consider the advice in
85+
the Style guide:
86+
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
87+
88+
2. Go through the breaking change process, as documented here:
89+
https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
90+
If you're going to do this, you will need to contact the relevant
91+
people responsible for the breaking test(s) (see the relevant .test
92+
files), help them fix their code, and update this repository to use
93+
the new version of their tests, in addition to the steps described
94+
on the wiki.
95+
96+
3. Remove the test in question. This is by far the least ideal
97+
solution. To go down this path, we must first establish that one of
98+
the following is true:
99+
100+
- the people listed as contacts for the test are not responsive.
101+
102+
- the test is poorly written (e.g. it contains a race condition or
103+
relies on assumptions that violate clearly documented API
104+
contracts), and the people listed as contacts are not willing to
105+
fix the test or accept fixes for the test.
106+
107+
- we have gone through the breaking change process cited above,
108+
but are unable to update the test accordingly (e.g. the people
109+
listed as contacts are not willing to work with us to update
110+
their code).

registry/DanTup_tiler.test

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
contact=danny@tuppeny.com
2+
fetch=git clone https://github.com/DanTup/tiler.git tests
3+
fetch=git -C tests checkout a448ce4781f02d11fbf9c310502a6b4382809ecc
4+
fetch=git -C tests submodule init
5+
fetch=git -C tests submodule update --recursive --remote
6+
update=.
7+
test=flutter test test

registry/flutter_svg.test

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
contact=dfield@gmail.com
2+
fetch=git clone https://github.com/dnfield/flutter_svg.git tests
3+
fetch=git -C tests checkout a091088418dcb2acbbd103b0472f9dc309bfda2f
4+
update=.
5+
test=flutter test

registry/template.test

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# This is a template for adding tests to the flutter/tests registry.
2+
#
3+
# To submit a new batch of tests, copy this file to create a new file
4+
# in this directory, update it as described below, and submit it as a
5+
# new PR to this repository.
6+
#
7+
# New files should be named `foo.test` where `foo` is a description of
8+
# the project whose tests are being added. (The file named
9+
# `template.test` is skipped automatically.)
10+
#
11+
# Lines beginning with a "#" are ignored. Delete all such lines from
12+
# this template before submitting a new `.test` file (you may add new
13+
# comments if you have something to say that doesn't fit in one of the
14+
# meaningful lines). Also delete blank lines like the following:
15+
16+
# The contact lines must contain the e-mail addresses, one to a line,
17+
# of one or more people who will be able to help if a problem is found
18+
# with a test. People on this list should be able to respond to
19+
# e-mails within about 72 hours. If there is a problem with a test and
20+
# we cannot reach any of the people listed, then we will probably
21+
# remove the tests.
22+
contact=foo@example.com
23+
contact=bar@example.com
24+
25+
# The fetch line(s) must contain one or more commands that fetch the
26+
# tests into a new directory called "tests" in the current directory.
27+
# The example below clones the dnfield/flutter_svg GitHub repository
28+
# then checks out a specific revision.
29+
#
30+
# Our current policy is that all tests should be in public GitHub
31+
# repositories. (This way, we don't take on additional dependencies on
32+
# other services that may or not go down. We already depend on GitHub,
33+
# so if GitHub goes down it doesn't matter that we can't reach
34+
# contributed tests.) If you have some tests to contribute that are
35+
# not and will not be in GitHub, please contact ian@hixie.ch before
36+
# submitting a PR proposing to add such tests. We may make exceptions
37+
# if your tests are especially valuable.
38+
#
39+
# The checkout must always check out the exact same set of tests. Do
40+
# not check out the current master branch or some such; always fetch a
41+
# specific revision. You may regularly submit PRs that update the
42+
# revision being used.
43+
#
44+
# Each fetch directive is processed as follows: the value is split on
45+
# spaces (without honoring quotes or backslash escapes or anything
46+
# like that), then the first segment is used as the executable to ask
47+
# the operating system to run, and the rest are treated as arguments
48+
# to pass to that process.
49+
#
50+
# The commands specified on the "fetch" lines must return the exit
51+
# code 0 or else the tests will be skipped entirely. The "git" program
52+
# will be on the path.
53+
#
54+
# There must be at least one "fetch" line. (In fact currently there
55+
# must be at least two, and the first two must follow the pattern
56+
# below more or less exactly. Additional lines are allowed e.g. to
57+
# initialize and update submodules, but this should be rare. If this
58+
# is a problem, let us know in your PR.)
59+
fetch=git clone https://github.com/USERNAME/REPOSITORY.git tests
60+
fetch=git -C tests checkout 0123456789abcdef0123456789abcdef01234567
61+
62+
# The "update" lines specify paths (relative to the newly created
63+
# "tests" directory) in which to run a flutter command to
64+
# automatically update the code to fix breaking changes. The command
65+
# will be run in turn in each specified directory. The root can be
66+
# specified by using the path "." as that represents the "current"
67+
# directory. Multiple paths may be specified by specifying multiple
68+
# "update" lines.
69+
#
70+
# The command that updates the code may fail, in which case the tests
71+
# are considered to have failed. This helps catch cases where
72+
# automatically updating the code doesn't work.
73+
#
74+
# There must be at least one "update" line.
75+
update=.
76+
77+
# The test lines specify commands to run to execute the actual tests.
78+
# Each of these should return either 0 to indicate success, or a
79+
# non-zero exit code to report failure.
80+
#
81+
# The "flutter" tool and the "dart" program will be in the path.
82+
#
83+
# If there are multiple directories, create a shell script in the
84+
# repository to enter each directory and run the tests in that
85+
# directory, returning non-zero as soon as any set of tests fails. You
86+
# can have multiple such scripts. In the example below, we run
87+
# "flutter test" in the root of the repository, a "more_tests.sh"
88+
# shell script from that same directory (the "./" indicates it isn't a
89+
# command to search for on the path), and the "extra_tests.dart"
90+
# program in the "dev" directory, run via "dart".
91+
test=flutter test
92+
test=./more_tests.sh
93+
test=dart dev/extra_tests.dart
94+
95+
# To test your tests, check out the flutter/flutter repository and
96+
# then, from the root of that repository, run:
97+
# bin/flutter; time bin/cache/dart-sdk/bin/dart dev/customer_testing/run_tests.dart --repeat=100 <path>
98+
# ...where <path> is the path to this file. This should run with no
99+
# output, without failing, and should in total take less than 15
100+
# minutes (a hundred runs of your tests taking just a few seconds
101+
# each). If your tests take longer, mention this in your PR and we
102+
# will evaluate them to see how valuable they are (e.g. how unique and
103+
# different they are compared to other tests we already have). The
104+
# more valuable the tests, the more likely we are to accept them
105+
# despite them taking a long time to run.
106+
107+
# See also the `README.md` file in the root of this repository.

scripts/verify_tests_on_master.bat

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
@ECHO ON
2+
3+
REM Fetch Flutter.
4+
git clone https://github.com/flutter/flutter.git || GOTO :END
5+
CALL flutter\bin\flutter doctor -v || GOTO :END
6+
7+
@ECHO.
8+
@ECHO.
9+
10+
REM Now run the tests a bunch of times to try to find flakes (tests that sometimes pass
11+
REM even though they should be failing).
12+
CALL flutter\bin\cache\dart-sdk\bin\dart flutter\dev\customer_testing\run_tests.dart --repeat=15 --skip-template registry/*.test || GOTO :END
13+
14+
@ECHO.
15+
@ECHO.
16+
@ECHO Testing complete.
17+
GOTO :EOF
18+
19+
:END
20+
@ECHO.
21+
@ECHO.
22+
@ECHO Testing failed.
23+
EXIT /B 1

scripts/verify_tests_on_master.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
set -ex
2+
3+
# Fetch Flutter.
4+
git clone https://github.com/flutter/flutter.git
5+
flutter/bin/flutter doctor -v
6+
7+
# Now run the tests a bunch of times to try to find flakes (tests that sometimes pass
8+
# even though they should be failing).
9+
flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat=15 --skip-template registry/*.test

0 commit comments

Comments
 (0)