murdock: introduce 'TEST_ON_CI_BLACKLIST'

[cladmi]

Contribution description

Introduce a variable to set that a test is blacklisted.

This is a move toward enabling tests by default and adding a blacklisting
reason instead for a board instead of not whitelisting them which hides
the problem.

Currently, a test should be both whitelisted and blacklisted at the same
time to have a meaning. It is planned to whitelist all by default in
an upcoming pull request.

Testing procedure

TEST_ON_CI_BLACKLIST behaves properly

The tests/xtimer_now64_continuity and tests/xtimer_usleep must correctly run by murdock on samr21-xpro and nrf52dk, and not native.

https://ci.riot-os.org/RIOT-OS/RIOT/11743/bdd721fff476ae8ffe11b6240e7d0e0a6070999f/output/run_test/tests/xtimer_now64_continuity/nrf52dk:gnu.txt

And it is indeed not executed in native (it is in the compiler output for native).

https://ci.riot-os.org/RIOT-OS/RIOT/11743/bdd721fff476ae8ffe11b6240e7d0e0a6070999f/output/compile/tests/xtimer_now64_continuity/native:gnu.txt

It creates the .test file and test-on-ci-enabled has expected return value

export BOARD=native; RIOT_CI_BUILD=1 make -C tests/xtimer_usleep 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C
 $_; ls $_/bin/${BOARD}/.test
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:46: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
ls: cannot access 'tests/xtimer_usleep/bin/native/.test': No such file or directory
export BOARD=samr21-xpro; RIOT_CI_BUILD=1 make -C tests/xtimer_usleep 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabl
ed -C $_; ls $_/bin/${BOARD}/.test
tests/xtimer_usleep/bin/samr21-xpro/.test
export BOARD=nrf52dk; RIOT_CI_BUILD=1 make -C tests/xtimer_usleep 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -
C $_; ls $_/bin/${BOARD}/.test
tests/xtimer_usleep/bin/nrf52dk/.test

Previous behavior is preserved

Tests with TEST_ON_CI_WHITELIST += all still create the .test file and test-on-ci-enabled has expected return value

export BOARD=native; RIOT_CI_BUILD=1 make -C tests/bloom_bytes/ 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C $_; ls $_/bin/${BOARD}/.test
tests/bloom_bytes//bin/native/.test
export BOARD=samr21-xpro; RIOT_CI_BUILD=1 make -C tests/bloom_bytes/ 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C $_; ls $_/bin/${BOARD}/.test
tests/bloom_bytes//bin/samr21-xpro/.test

With TEST_ON_CI_WHITELIST = native only native does it

export BOARD=native; RIOT_CI_BUILD=1 make -C tests/libfixmath_unittests/ 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C $_; ls $_/bin/${BOARD}/.test
tests/libfixmath_unittests//bin/native/.test
export BOARD=samr21-xpro; RIOT_CI_BUILD=1 make -C tests/libfixmath_unittests/ 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C $_; ls $_/bin/${BOARD}/.test
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:46: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
ls: cannot access 'tests/libfixmath_unittests//bin/samr21-xpro/.test': No such file or directory

If empty, tests are disabled

export BOARD=native; RIOT_CI_BUILD=1 make -C examples/hello-world/ 2>/dev/null >/dev/null ; make --no-print-directory test-on-ci-enabled -C $_; ls $_/bin/${BOARD}/.test
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:46: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
ls: cannot access 'examples/hello-world//bin/native/.test': No such file or directory

Issues/PRs references

• Split out ouf: makefiles/murdock.inc.mk: change policy to run tests by default makefiles/murdock.inc.mk: change policy to run tests by default #11680
• Asked in tests/pkg_u8g2: run the test in CI #11687

[cladmi]

I disable test cache as otherwise, murdock does not include the tests results.
I want to check that indeed tests are run.

[cladmi]

I added a usage for TEST_ON_CI_BLACKLIST and a testing procedure.
This should also allow running the changed tests on nrf52dk.

[kaspar030]

This is the diff between tests run for last nightlies and this PR:

[kaspar@ng ~/tmp/foo]$ diff pr master
88a89,92
> tests/gnrc_ipv6_fwd_w_sub/nrf52dk:gnu
> tests/gnrc_ipv6_fwd_w_sub/nrf52dk:llvm
> tests/gnrc_ipv6_fwd_w_sub/samr21-xpro:gnu
> tests/gnrc_ipv6_fwd_w_sub/samr21-xpro:llvm
245a250,253
> tests/pkg_nanocbor/nrf52dk:gnu
> tests/pkg_nanocbor/nrf52dk:llvm
> tests/pkg_nanocbor/samr21-xpro:gnu
> tests/pkg_nanocbor/samr21-xpro:llvm
403,404d410
< tests/xtimer_now64_continuity/nrf52dk:gnu
< tests/xtimer_now64_continuity/nrf52dk:llvm
419,420d424
< tests/xtimer_usleep/nrf52dk:gnu
< tests/xtimer_usleep/nrf52dk:llvm

There's something wrong with the base branch of the CI build. It was merged into "eab0a8864250001567cb0cbfc1c5562ba09a3af2", which is quite old. I'm investigating.

[kaspar030]

I think I fell over a browser-cached, previous result. Correct diff is:

[kaspar@ng ~/tmp/foo]$ diff master pr
410a411,412
> tests/xtimer_now64_continuity/nrf52dk:gnu
> tests/xtimer_now64_continuity/nrf52dk:llvm
424a427,428
> tests/xtimer_usleep/nrf52dk:gnu
> tests/xtimer_usleep/nrf52dk:llvm

[kaspar030]

ACK.

[cladmi]

Thank you for the review.