Skip to content

Commit

Permalink
Gathering information about test filter files into a README.md file.
Browse files Browse the repository at this point in the history
BUG=

Review-Url: https://codereview.chromium.org/2472153002
Cr-Commit-Position: refs/heads/master@{#430623}
  • Loading branch information
anforowicz authored and Commit bot committed Nov 8, 2016
1 parent 552106f commit f2fe3f7
Show file tree
Hide file tree
Showing 4 changed files with 90 additions and 2 deletions.
2 changes: 2 additions & 0 deletions base/test/launcher/test_launcher.cc
Original file line number Diff line number Diff line change
Expand Up @@ -842,6 +842,8 @@ bool TestLauncher::Init() {
return false;
}

// Parse the file contents (see //testing/buildbot/filters/README.md
// for file syntax and other info).
std::vector<std::string> filter_lines = SplitString(
filter, "\n", base::TRIM_WHITESPACE, base::SPLIT_WANT_ALL);
for (const std::string& filter_line : filter_lines) {
Expand Down
1 change: 1 addition & 0 deletions base/test/launcher/unit_test_launcher.cc
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ void PrintUsage() {
" --test-launcher-filter-file=PATH\n"
" Like --gtest_filter, but read the test filter from PATH.\n"
" One pattern per line; lines starting with '-' are exclusions.\n"
" See also //testing/buildbot/filters/README.md file.\n"
"\n"
" --test-launcher-batch-limit=N\n"
" Sets the limit of test batch to run in a single process to N.\n"
Expand Down
4 changes: 2 additions & 2 deletions build/android/test_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,8 @@ def AddGTestOptions(parser):
filter_group.add_argument('--gtest-filter-file', dest='test_filter_file',
type=os.path.realpath,
help='Path to file that contains googletest-style '
'filter strings. (Lines will be joined with '
'":" to create a single filter string.)')
'filter strings. See also '
'//testing/buildbot/filters/README.md.')

AddDeviceOptions(parser)
AddCommonOptions(parser)
Expand Down
85 changes: 85 additions & 0 deletions testing/buildbot/filters/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Test filter files

## Summary

This directory contains files that list tests that are not yet ready to run in a
particular mode. For example - the `site-per-process.browser_tests.filter` file
lists tests that should be excluded when running `browser_tests` in
`--site-per-process` mode.

## File syntax

Contents of test filter files follow the syntax below:

- Empty lines are ignored
- Lines starting with the '#' character are treated as comments and ignored.
- All other lines specify a single [test name pattern][gtest_filter].
- Negative patterns: Patterns prefixed with the '-' character specify
tests to exclude from a test run.
- Positive patterns: All other patterns specify tests to include in a test
run.

A test will be executed if and only if it matches any of the positive patterns
but does not match any of the negative patterns (please see documentation of
[--gtest_filter][gtest_filter] for more details and examples).

Please see [https://crbug.com/587527] for discussion how "positive" and
"negative" test patterns should be combined in presence of both the
`--gtest_filter` and the `--test-launcher-filter-file` command line flags.

Example test filter file for excluding a set of tests:

```test.filter
# crbug.com/417518: Get tests working w/ --site-per-process
-BrowserTest.OtherRedirectsDontForkProcess
-ChromeRenderProcessHostTest.*
-ReferrerPolicyTest.HttpsRedirect
# crbug.com/448592: Get extension browsertests working w/ --site-per-process
-IsolatedAppTest.CookieIsolation
-IsolatedAppTest.CrossProcessClientRedirect
-IsolatedAppTest.IsolatedAppProcessModel
-IsolatedAppTest.SubresourceCookieIsolation
```

## Usage

When running tests on desktop platforms, the test filter file can be specified
using `--test-launcher-filter-file` command line flag. Example test invocation:

```bash
$ out/dbg/content_browsertests \
--site-per-process \
--test-launcher-filter-file=testing/buildbot/filters/site-per-process.content_browsertests.filter
```

When running tests on Android, the test filter file can be specified using
`--gtest-filter-file` command line flag. Example test invocation:

```bash
$ out/android/bin/run_content_browsertests \
--test-arguments=--site-per-process \
--gtest-filter-file=testing/buildbot/filters/site-per-process.content_browsertests.filter
```

## Applicability

Test filter files described here are currently only supported for gtest-based
tests.

For excluding layout tests when running with a particular command line flag, see
`third_party/WebKit/LayoutTests/FlagExpectations/README.txt`.

## Adding new test filter files

Please use the following conventions when naming the new file:

- Please include the name of the test executable (e.g.
`content_browsertests`).
- Please use `.filter` suffix.
- Feel free to add other relevant things into the file name (e.g. the mode the
file applies to - for example `site-per-process`).

When adding a new file, please update `//testing/buildbot/filters/BUILD.gn`.

[gtest_filter]: https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#running-a-subset-of-the-tests

0 comments on commit f2fe3f7

Please sign in to comment.