Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

boards/microbit: tracker for CI blacklist due to test failing with qemu #17894

Open
3 of 47 tasks
kaspar030 opened this issue Apr 1, 2022 · 5 comments
Open
3 of 47 tasks

boards/microbit: tracker for CI blacklist due to test failing with qemu #17894

kaspar030 opened this issue Apr 1, 2022 · 5 comments
Assignees
@kfessel
Labels
Community: help wanted The contributors require help from other members of the community Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation

Comments

@kaspar030
Copy link
Contributor

Description

#17434 enables CI to test for microbit emulated with qemu.
Some tests fail. To get it in, they have been CI blacklisted for further investigation.

IMO, they all need to be checked if the failure is related to qemu not emulating hardware, or qemu not being exact enough (timing off?), or something wrong with the setup (pyterm seems to make some tests fail?). So I dump the list here.

As a reminder, if qemu is installed (qemu-system-arm needs to be available), a test can be run like this:

EMULATE=1 BOARD=microbit make -Ctests/fmt_print all test

Missing rtt:

  • tests/periph_rtt_min

Not exact enough:

  • tests/ztimer_overhead -> timing off
  • tests/thread_flags -> timing off

Unknown:

  • tests/evtimer_mbox
  • tests/driver_nrfmin
  • tests/gnrc_ipv6_fwd_w_sub
  • tests/heap_cmd
  • tests/evtimer_msg
  • tests/netstats_l2
  • tests/event_wait_timeout_ztimer
  • tests/mtd_raw
  • tests/evtimer_underflow
  • tests/periph_gpio
  • tests/gnrc_netif_ipv6_wait_for_global_address
  • tests/gnrc_sixlowpan_frag_sfr
  • tests/periph_flashpage
  • tests/bench_xtimer_load
  • tests/gnrc_tx_sync
  • tests/periph_wdt
  • tests/log_color -> looks like a term issue
  • tests/lwip_sock_udp
  • tests/test_tools
  • tests/struct_tm_utility
  • tests/ps_schedstatistics
  • tests/sys_atomic_utils
  • tests/pthread_cooperation
  • tests/xtimer_usleep
  • tests/lwip_sock_ip
  • tests/turo_txt
  • tests/xtimer_now64_continuity
  • tests/sys_sched_round_robin
  • tests/xtimer_rmutex_lock_timeout
  • tests/xtimer_periodic_wakeup
  • tests/ztimer_rmutex_lock_timeout
  • tests/ztimer_periodic
  • tests/periph_rtt
  • tests/trickle
  • tests/posix_semaphore
  • tests/xtimer_mutex_lock_timeout
  • tests/xtimer_hang
  • tests/shell
  • tests/xtimer_usleep_short
  • tests/ztimer_underflow
  • tests/posix_sleep
  • tests/ztimer_xsec
  • tests/lwip_sock_tcp
  • tests/pkg_umorse

Steps to reproduce the issue

Expected results

Actual results

Versions
@kaspar030 kaspar030 mentioned this issue Apr 1, 2022
Merged
@kaspar030 kaspar030 added Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation Community: help wanted The contributors require help from other members of the community labels Apr 1, 2022
@kfessel
Copy link
Contributor

kfessel commented Apr 11, 2022 •
edited
Loading

seems like it is not unlikely that these emulates tests fail due to timing issues

image

(I ran the tests at my laptop and they succeeded)

-- running on worker comsys thread 4, build number 17514.
make: Entering directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout'
Building application "tests_event_wait_timeout" for "microbit" with MCU "nrf51".

sha1sum /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout/tests/01-run.py /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf > /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/test-input-hash.sha1
   text	   data	    bss	    dec	    hex	filename
  12372	    176	   3428	  15976	   3e68	/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf
make: Leaving directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout'
make: Entering directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout'
r
/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/dist/tools/emulator/term.sh qemu microbit /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/dist/tools/pyterm/pyterm '-p "/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart" -b "115200" ' /tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart /tmp/riot_tests_event_wait_timeout_microbit.jTfrf 
qemu-system-arm -machine microbit -device loader,file=/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf -serial  unix:/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart_socket,server=on,wait=off -monitor unix:/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/mon_socket,server=on,wait=off -nographic
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2022-04-11 17:21:10,326 # Connect to serial port /tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart
Welcome to pyterm!
Type '/exit' to exit.
2022-04-11 17:21:11,329 # READY
s
2022-04-11 17:21:11,382 # START
2022-04-11 17:21:11,383 # main(): This is RIOT! (Version: buildtest)
2022-04-11 17:21:11,384 # [START] event_wait_timeout test application.
2022-04-11 17:21:11,384 # 
Timeout in expect script at "child.expect_exact(u"[SUCCESS]")" (tests/event_wait_timeout/tests/01-run.py:15)

make: *** [/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/makefiles/tests/tests.inc.mk:22: test] Error 1
make: Leaving directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout'
{"build/": 940}

-- running on worker mobi8.inet.haw-hamburg.de thread 16, build number 9937.
make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
Building application "tests_cb_mux_bench" for "microbit" with MCU "nrf51".

sha1sum /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench/tests/01-run.py /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf > /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/test-input-hash.sha1
   text	   data	    bss	    dec	    hex	filename
  11216	    168	   2800	  14184	   3768	/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf
make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
Building application "tests_cb_mux_bench" for "microbit" with MCU "nrf51".

sha1sum /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench/tests/01-run.py /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf > /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/test-input-hash.sha1
   text	   data	    bss	    dec	    hex	filename
  11216	    168	   2800	  14184	   3768	/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf
make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
r
/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/dist/tools/emulator/term.sh qemu microbit /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/dist/tools/pyterm/pyterm '-p "/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart" -b "115200" ' /tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart /tmp/riot_tests_cb_mux_bench_microbit.J0ak2 
qemu-system-arm -machine microbit -device loader,file=/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf -serial  unix:/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart_socket,server=on,wait=off -monitor unix:/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/mon_socket,server=on,wait=off -nographic
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2022-04-11 17:08:53,033 # Connect to serial port /tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart
Welcome to pyterm!
Type '/exit' to exit.
2022-04-11 17:08:54,042 # READY
s
2022-04-11 17:08:54,094 # START
2022-04-11 17:08:54,096 # main(): This is RIOT! (Version: buildtest)
2022-04-11 17:08:54,097 # cb_mux benchmark application
2022-04-11 17:08:54,098 # Populating cb_mux list with 20 items
2022-04-11 17:08:54,099 # Finding the last list entry
2022-04-11 17:08:54,100 # List walk time: 274 us
2022-04-11 17:08:54,101 # Walk time greater than threshold of 200 us
2022-04-11 17:08:54,102 # [FAILURE]
2022-04-11 17:08:54,104 # { "threads": [{ "name": "main", "stack_size": 1536, "stack_used": 436 }]}
Timeout in expect script at "child.expect(r"Walk time less than threshold of \d+ us")" (tests/cb_mux_bench/tests/01-run.py:18)

make: *** [/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/makefiles/tests/tests.inc.mk:22: test] Error 1
make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench'
{"build/": 920}

@kaspar030
Copy link
Contributor Author

yes, very likely. do you want to open a blacklist PR?

@kfessel
Copy link
Contributor

kfessel commented Apr 11, 2022 •
edited
Loading

I am not sure since it does not seem happen often (only one other PR shows this atm "PR #16771 build at 11/04/2022, 13:27:52") (maybe this happens when the building server has to much other load?)
On the other hand ci.riot / murdock shows only the latest run of a PR.
Maybe you have a better view at the past failing builds? (could we ignore the results but still run the test?)

Some of these timeouts might be solved by setting RIOT_TEST_TIMEOUT for emulated microbit

This was referenced Apr 12, 2022
Merged
Closed
Merged
@kfessel
Copy link
Contributor

kfessel commented Apr 12, 2022

https://ci.riot-os.org/RIOT-OS/RIOT/17096/c7638d84ae5a709d6634ffe81dbce6828ecbab3f/output/compile/tests/pbkdf2/microbit:gnu.txt
https://ci.riot-os.org/RIOT-OS/RIOT/16771/5a6b007467d22bee4ec7278782d272d8df7cd4d5/output/compile/tests/event_wait_timeout/microbit:gnu.txt
https://ci.riot-os.org/RIOT-OS/RIOT/16771/5a6b007467d22bee4ec7278782d272d8df7cd4d5/output/compile/tests/pbkdf2/microbit:gnu.txt

@kfessel
Copy link
Contributor

kfessel commented Apr 13, 2022

https://qemu.readthedocs.io/en/latest/system/arm/nrf.html

maybe we should make the emulated microbit a seperate board (with a different feature set)

bors bot added a commit that referenced this issue Jan 5, 2023
@bors @kfessel @maribu
Merge #17919 #19094
4a3ee04 
17919: board/microbit: increase testtimeout for emulated testing r=benpicco a=kfessel

### Contribution description

this increases the timeout for tests run by murdock on emulated microbit (accounting for startup of qemu)

### Testing procedure

murdock

### Issues/PRs references

#17894 

19094: dist/tools/doccheck: Reduce noise r=benpicco a=maribu

### Contribution description

This adds central documentation of driver initialization parameters in `driver/foo/include/foo_params.h` where missing for drivers used by the Hamilton board. In addition, the macros to overwrite the default parameters are no longer expected to be documented by adding generic exclude patterns. The reasoning is that having the macros documented once is sufficient.

### Testing procedure

Static tests should pass

### Issues/PRs references

Required to get #19090 in

Co-authored-by: Karl Fessel <karl.fessel@ovgu.de>
Co-authored-by: Marian Buschsieweke <marian.buschsieweke@ovgu.de>
bors bot added a commit that referenced this issue Jan 5, 2023
@bors @kfessel
Merge #17919
170dd03 
17919: board/microbit: increase testtimeout for emulated testing r=benpicco a=kfessel

### Contribution description

this increases the timeout for tests run by murdock on emulated microbit (accounting for startup of qemu)

### Testing procedure

murdock

### Issues/PRs references

#17894 

Co-authored-by: Karl Fessel <karl.fessel@ovgu.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kfessel kfessel

Labels
Community: help wanted The contributors require help from other members of the community Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kfessel @kaspar030