Skip to content

Commit

Permalink
nightly: document --skip-build test flag plus some other minor updates
Browse files Browse the repository at this point in the history
Issue: near#3186
Issue: near#4726
  • Loading branch information
mina86 committed Nov 25, 2021
1 parent feb0acd commit ac867e5
Show file tree
Hide file tree
Showing 2 changed files with 28 additions and 25 deletions.
51 changes: 27 additions & 24 deletions nightly/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,34 +14,28 @@ with a hash are ignored. The rest either specifies a test to run or
a list file to include. The general syntax of a line defining a test
is:

<category> [--timeout=<timeout>] [--release] [--remote] <args>... [--features <features>]
<category> [--skip-build] [--timeout=<timeout>] [--release] [--remote] <args>... [--features <features>]

`<category>` specifies the category of the test and can be one of
`pytest`, `mocknet` or `expensive`. The meaning of `<args>` depends
on the test category.
`<category>` specifies the category of the test and can be `pytest` or
`expensive`. The meaning of `<args>` depends on the test category.

### pytest and mocknet tests
### pytest tests

The `pytest` and `mocknet` tests are run by executing a file within
`pytest/tests` directory. The file is executed via `python`
interpreter (and confusingly not via `pytest` command) so it must
actually run the tests rather than only defining them as test
functions.
The `pytest` tests are run by executing a file within `pytest/tests`
directory. The file is executed via `python` interpreter (and
confusingly not via `pytest` command) so it must actually run the
tests rather than only defining them as test functions.

In the test specification path to the file needs to be given
(excluding the `pytest/tests` prefix) and anything that follows is
passed as arguments to the script. For example:

pytest sanity/lightclnt.py
pytest sanity/state_sync_routed.py manytx 115
mocknet mocknet/sanity.py

The difference between `pytest` and `mocknet` tests is that the latter
skips the build stage since it only ever talks with the existing
mocknet cluster and does not need the `neard` binary to be run. Other
then that, `mocknet` test could be run as `pytest` and they would work
the same way (they would just waste time and resources executing the
build).
Note: NayDuck also handles `mocknet` test category. It is now
deprecated and is treated like `pytest` with `--skip-build` flag
implicitly set.

### expensive tests

Expand All @@ -68,11 +62,17 @@ match exactly an the test binary is called with `--exact` argument.

As mentioned, there are additional arguments that can go between the
test category and the test specification arguments. Those are
`--timeout`, `--release` and `--remote`.
`--skip-build`, `--timeout`, `--release` and `--remote`.

`--timeout=<timeout>` which specifies the time after which the test
will be stopped and considered failed. `<timeout>` is an integer with
an optional `s`, `m` or `h` suffix. If no suffix is given, `s` is
`--skip-build` causes build step to be skipped for the test. This
means that the test doesn’t have access to build artefacts (located in
`target/debug` or `target/release`) but also doesn’t need to wait for
the build to finish and thus can start faster. For example, the flag
is used with fuzzing tests which need to build their own code.

`--timeout=<timeout>` specifies the time after which the test will be
stopped and considered failed. `<timeout>` is an integer with an
optional `s`, `m` or `h` suffix. If no suffix is given, `s` is
assumed. The default timeout is three minutes. For example, the
following increases timeout for a test to four minutes:

Expand All @@ -91,8 +91,12 @@ honest I can’t vouch whether it even works.
Lastly, at the end of the test specification line additional features
can be given in the form of `--features <features>` arguments.
Similarly to `--release`, this results in given features being enabled
in builds. Note that the `test_features` feature is always enabled so
there's no need to specify it explicitly.
in builds. Note that the `test_features` and `rosetta_rpc` features
are always enabled so there's no need to specify it explicitly.

Note that with `--skip-build` switch the `--release` and `--features`
flags are essentially ignored since they only affect the build and are
not passed to the test.

### Include directive

Expand All @@ -106,7 +110,6 @@ For example, `nightly.txt` file may just look as follows:
./sandbox.txt
./pytest.txt
./expensive.txt
./mocknet.txt

with individual tests listed in each of the referenced files. This
makes the files more tidy.
Expand Down
2 changes: 1 addition & 1 deletion nightly/fuzzing.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
mocknet --timeout=7200 runtime/fuzz.py
pytest --skip-build --timeout=2h runtime/fuzz.py

0 comments on commit ac867e5

Please sign in to comment.