Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dist/pythonlibs/testrunner: reset before term #12862

Merged
merged 2 commits into from
Dec 19, 2019
Merged

dist/pythonlibs/testrunner: reset before term #12862

merged 2 commits into from
Dec 19, 2019

Conversation

fjmolinas
Copy link
Contributor

Contribution description

For some boardw make reset is only possible if a serial connection
is not already open or its execution might disrupt it. This
causes some tests to fail since before running a test the board
is reset.

make reset is currently used as a synchronization mechanism between
the application and the test script. With test_utils_interactive_sync
this is no longer needed so call make reset before cleanterm instead
of after so avoid using it for those boards.

With the wide usage of test_utils_interactive_sync #12448 is not needed, only
a handful of tests remain that do not use it, and most of same need sudo.
This will be addressed in a followup PR.

Testing procedure

  • This PR
python dist/tools/compile_and_test_for_board/compile_and_test_for_board.py -j0 . iotlab-m3 . --with-test-only 
#### iotlab-m3/failuresummary.md                          
Failures during test:                                                                                       
- [examples/suit_update](examples/suit_update/test.failed)                                                  
- [tests/driver_hd44780](tests/driver_hd44780/test.failed)                                                  
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)                                                    
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)                                          
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)                                                      
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)                                                    
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)                                                              
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
python dist/tools/compile_and_test_for_board/compile_and_test_for_board.py -j0 . remote-revb . --with-test-only 
#### remote-revb/failuresummary.md                                                                               
Failures during test:                                                   
- [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)      
- [tests/driver_hd44780](tests/driver_hd44780/test.failed)    
- [tests/driver_my9221](tests/driver_my9221/test.failed)              
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)      
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)      
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)        
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)                            
- [tests/periph_timer](tests/periph_timer/test.failed)        
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)            
- [tests/pkg_qdsa](tests/pkg_qdsa/test.failed)
python dist/tools/compile_and_test_for_board/compile_and_test_for_board.py -j0 . native . --with-test-only 
Failures during test:
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)                                                    
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)                                          
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)                                                      
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed) 
- [tests/netstats_l2](tests/netstats_l2/test.failed)
  • master
python dist/tools/compile_and_test_for_board/compile_and_test_for_board.py -j0 . iotlab-m3 . --with-test-only 
#### iotlab-m3/failuresummary.md                          
Failures during test:                                                                                       
- [examples/suit_update](examples/suit_update/test.failed)
- [tests/driver_hd44780](tests/driver_hd44780/test.failed)
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)
- [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed)
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
python dist/tools/compile_and_test_for_board/compile_and_test_for_board.py -j0 . remote-revb . --with-test-only 
#### remote-revb/failuresummary.md                                                                               
Failures during test:
- [tests/bench_msg_pingpong](tests/bench_msg_pingpong/test.failed)                                                                       
- [tests/bench_mutex_pingpong](tests/bench_mutex_pingpong/test.failed)                                                                   
- [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed)                                                               
- [tests/bench_sched_nop](tests/bench_sched_nop/test.failed)                                                                             
- [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed)                                                               
- [tests/bench_thread_flags_pingpong](tests/bench_thread_flags_pingpong/test.failed)                                                     
- [tests/bench_thread_yield_pingpong](tests/bench_thread_yield_pingpong/test.failed)                                                     
- [tests/bitarithm_timings](tests/bitarithm_timings/test.failed)                                                                         
- [tests/blob](tests/blob/test.failed)                                                                                                   
- [tests/bloom_bytes](tests/bloom_bytes/test.failed)                                                                                     
- [tests/build_system_cflags_spaces](tests/build_system_cflags_spaces/test.failed)                                                       
- [tests/buttons](tests/buttons/test.failed)                                                                                             
- [tests/cb_mux](tests/cb_mux/test.failed)                                                                                               
- [tests/cb_mux_bench](tests/cb_mux_bench/test.failed)                                                                                   
- [tests/cond_order](tests/cond_order/test.failed)                                                                                       
- [tests/cpp11_condition_variable](tests/cpp11_condition_variable/test.failed)                                                           
- [tests/cpp11_mutex](tests/cpp11_mutex/test.failed)                                                                                     
- [tests/cpp11_thread](tests/cpp11_thread/test.failed)                                                                                   
- [tests/cpp_ctors](tests/cpp_ctors/test.failed)                                                                                         
- [tests/devfs](tests/devfs/test.failed)                                                                                                 
- [tests/driver_ds1307](tests/driver_ds1307/test.failed)                                                                                 
- [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)                                                                     
- [tests/driver_hd44780](tests/driver_hd44780/test.failed)                                                                               
- [tests/driver_my9221](tests/driver_my9221/test.failed)                                                                                 
- [tests/embunit](tests/embunit/test.failed)                                                                                             
- [tests/event_wait_timeout](tests/event_wait_timeout/test.failed)                                                                       
- [tests/events](tests/events/test.failed)                                                                                               
- [tests/evtimer_msg](tests/evtimer_msg/test.failed)                                                                                     
- [tests/evtimer_underflow](tests/evtimer_underflow/test.failed)                                                                         
- [tests/float](tests/float/test.failed)                                                                                                 
- [tests/fmt_print](tests/fmt_print/test.failed)                                                                                         
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)                                                                                 
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)                                                                       
- [tests/gnrc_ipv6_fwd_w_sub](tests/gnrc_ipv6_fwd_w_sub/test.failed)                                                                     
- [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.failed)                                                                                 
- [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.failed)                                                                         
- [tests/gnrc_ndp](tests/gnrc_ndp/test.failed)                                                                                           
- [tests/gnrc_netif](tests/gnrc_netif/test.failed)                                                                                       
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)                                                                                   
- [tests/gnrc_sixlowpan](tests/gnrc_sixlowpan/test.failed)                                                                               
- [tests/gnrc_sixlowpan_frag](tests/gnrc_sixlowpan_frag/test.failed)                                                                     
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)                                                                                 
- [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.failed)                                                                                   
- [tests/gnrc_sock_neterr](tests/gnrc_sock_neterr/test.failed)                                                                           
- [tests/gnrc_sock_udp](tests/gnrc_sock_udp/test.failed)                                                                                 
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)                                                                                           
- [tests/heap_cmd](tests/heap_cmd/test.failed)                                                                                           
- [tests/irq](tests/irq/test.failed)                                                                                                     
- [tests/isr_yield_higher](tests/isr_yield_higher/test.failed)                                                                           
- [tests/l2util](tests/l2util/test.failed)                                                                                               
- [tests/libc_newlib](tests/libc_newlib/test.failed)                                                                                     
- [tests/log_color](tests/log_color/test.failed)                                                                                         
- [tests/log_printfnoformat](tests/log_printfnoformat/test.failed)                                                                       
- [tests/lwip_sock_ip](tests/lwip_sock_ip/test.failed)                                                                                   
- [tests/lwip_sock_tcp](tests/lwip_sock_tcp/test.failed)                                                                                 
- [tests/lwip_sock_udp](tests/lwip_sock_udp/test.failed)                                                                                 
- [tests/malloc](tests/malloc/test.failed)                                                                                               
- [tests/memarray](tests/memarray/test.failed)                                                                                           
- [tests/msg_avail](tests/msg_avail/test.failed)
- [tests/msg_send_receive](tests/msg_send_receive/test.failed)
- [tests/msg_try_receive](tests/msg_try_receive/test.failed)
- [tests/mutex_order](tests/mutex_order/test.failed)
- [tests/mutex_unlock_and_sleep](tests/mutex_unlock_and_sleep/test.failed)
- [tests/netdev_test](tests/netdev_test/test.failed)
- [tests/nhdp](tests/nhdp/test.failed)
- [tests/od](tests/od/test.failed)
- [tests/periph_cpuid](tests/periph_cpuid/test.failed)
- [tests/periph_gpio](tests/periph_gpio/test.failed)
- [tests/periph_hwrng](tests/periph_hwrng/test.failed)
- [tests/periph_timer](tests/periph_timer/test.failed)
- [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed)
- [tests/pipe](tests/pipe/test.failed)
- [tests/pkg_c25519](tests/pkg_c25519/test.failed
- [tests/pkg_cayenne-lpp](tests/pkg_cayenne-lpp/test.failed)
- [tests/pkg_cifra](tests/pkg_cifra/test.failed)
- [tests/pkg_cn-cbor](tests/pkg_cn-cbor/test.failed)
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
- [tests/pkg_hacl](tests/pkg_hacl/test.failed)                                             
- [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed)
- [tests/pkg_jsmn](tests/pkg_jsmn/test.failed)
- [tests/pkg_libb2](tests/pkg_libb2/test.failed)
- [tests/pkg_libcoap](tests/pkg_libcoap/test.failed)
- [tests/pkg_libcose](tests/pkg_libcose/test.failed)
- [tests/pkg_libfixmath](tests/pkg_libfixmath/test.failed)
- [tests/pkg_libfixmath_unittests](tests/pkg_libfixmath_unittests/test.failed)
- [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed)
- [tests/pkg_littlefs](tests/pkg_littlefs/test.failed)
- [tests/pkg_lora-serialization](tests/pkg_lora-serialization/test.failed)
- [tests/pkg_micro-ecc](tests/pkg_micro-ecc/test.failed)
- [tests/pkg_micro-ecc-with-hwrng](tests/pkg_micro-ecc-with-hwrng/test.failed)
- [tests/pkg_minmea](tests/pkg_minmea/test.failed)
- [tests/pkg_monocypher](tests/pkg_monocypher/test.failed)
- [tests/pkg_nanocbor](tests/pkg_nanocbor/test.failed)
- [tests/pkg_nanopb](tests/pkg_nanopb/test.failed)
- [tests/pkg_qdsa](tests/pkg_qdsa/test.failed)
- [tests/pkg_relic](tests/pkg_relic/test.failed)
- [tests/pkg_spiffs](tests/pkg_spiffs/test.failed)
- [tests/pkg_tiny-asn1](tests/pkg_tiny-asn1/test.failed)
- [tests/pkg_tinycbor](tests/pkg_tinycbor/test.failed)
- [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
- [tests/pkg_u8g2](tests/pkg_u8g2/test.failed)
- [tests/pkg_ubasic](tests/pkg_ubasic/test.failed)
- [tests/pkg_ucglib](tests/pkg_ucglib/test.failed)
- [tests/pkg_umorse](tests/pkg_umorse/test.failed)
- [tests/posix_semaphore](tests/posix_semaphore/test.failed)
- [tests/posix_time](tests/posix_time/test.failed)
- [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
- [tests/pthread](tests/pthread/test.failed)
- [tests/pthread_barrier](tests/pthread_barrier/test.failed)
- [tests/pthread_cleanup](tests/pthread_cleanup/test.failed)
- [tests/pthread_condition_variable](tests/pthread_condition_variable/test.failed)
- [tests/pthread_cooperation](tests/pthread_cooperation/test.failed)
- [tests/pthread_flood](tests/pthread_flood/test.failed)
- [tests/pthread_rwlock](tests/pthread_rwlock/test.failed)
- [tests/pthread_tls](tests/pthread_tls/test.failed)
- [tests/riotboot_hdr](tests/riotboot_hdr/test.failed)
- [tests/rmutex](tests/rmutex/test.failed)
- [tests/rmutex_cpp](tests/rmutex_cpp/test.failed)
- [tests/rng](tests/rng/test.failed)
- [tests/sched_testing](tests/sched_testing/test.failed)
- [tests/shell](tests/shell/test.failed)
- [tests/ssp](tests/ssp/test.failed)
- [tests/stdin](tests/stdin/test.failed)
- [tests/struct_tm_utility](tests/struct_tm_utility/test.failed)
- [tests/sys_crypto](tests/sys_crypto/test.failed)
- [tests/test_tools](tests/test_tools/test.failed)
- [tests/thread_basic](tests/thread_basic/test.failed)
- [tests/thread_cooperation](tests/thread_cooperation/test.failed)
- [tests/thread_exit](tests/thread_exit/test.failed)
- [tests/thread_flags](tests/thread_flags/test.failed)
- [tests/thread_flags_xtimer](tests/thread_flags_xtimer/test.failed)
- [tests/thread_flood](tests/thread_flood/test.failed)
- [tests/thread_msg](tests/thread_msg/test.failed)
- [tests/thread_msg_block_race](tests/thread_msg_block_race/test.failed)
- [tests/thread_msg_block_w_queue](tests/thread_msg_block_w_queue/test.failed)
- [tests/thread_msg_block_wo_queue](tests/thread_msg_block_wo_queue/test.failed)
- [tests/thread_msg_seq](tests/thread_msg_seq/test.failed)
- [tests/thread_race](tests/thread_race/test.failed)
- [tests/trickle](tests/trickle/test.failed)
- [tests/unittests](tests/unittests/test.failed)
- [tests/xtimer_hang](tests/xtimer_hang/test.failed)
- [tests/xtimer_msg](tests/xtimer_msg/test.failed)
- [tests/xtimer_msg_receive_timeout](tests/xtimer_msg_receive_timeout/test.failed)
- [tests/xtimer_mutex_lock_timeout](tests/xtimer_mutex_lock_timeout/test.failed)
- [tests/xtimer_now64_continuity](tests/xtimer_now64_continuity/test.failed)
- [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
- [tests/xtimer_remove](tests/xtimer_remove/test.failed)
- [tests/xtimer_reset](tests/xtimer_reset/test.failed)
- [tests/xtimer_usleep](tests/xtimer_usleep/test.failed)
- [tests/xtimer_usleep_short](tests/xtimer_usleep_short/test.failed)

NOTE: - [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed) is not fixed by this PR, but my branch didn't have the new test at the time I ran the tests.

Issues/PRs references

#12448
Alternative to #12520

@fjmolinas fjmolinas added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework labels Dec 3, 2019
@fjmolinas fjmolinas requested a review from aabadie December 3, 2019 14:05
@fjmolinas fjmolinas mentioned this pull request Dec 3, 2019
Merged
@fjmolinas
Copy link
Contributor Author

@aabadie I added a fixup that only rests after term if asked explicitly, there is no need for this anymore with #12867 merged.

@aabadie
Copy link
Contributor

aabadie commented Dec 12, 2019

I would be interested to also have test results with an arduino-uno (a really slow one).

@aabadie
Copy link
Contributor

aabadie commented Dec 12, 2019

Otherwise this PR looks good. The improvement with the remote-revb tests is impressive. Good job!

@fjmolinas fjmolinas mentioned this pull request Dec 13, 2019
Merged
@fjmolinas fjmolinas force-pushed the pr_dont_reset_sync branch 2 times, most recently from 07c69c2 to 0905164 Compare December 15, 2019 18:49
@fjmolinas
Copy link
Contributor Author

rebased to get #12941 that fixes false compilation errors.

@fjmolinas
Copy link
Contributor Author

I hade a failure in tests/sys_arduino, not sure why yet.

Failures during test:
- [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed)
- [tests/sys_arduino](tests/sys_arduino/test.failed)
- [tests/trickle](tests/trickle/test.failed)

@fjmolinas
Copy link
Contributor Author

I hade a failure in tests/sys_arduino, not sure why yet.

I think the terminal takes some time to be setup.. will re-test with the physical board.

@fjmolinas fjmolinas mentioned this pull request Dec 16, 2019
Merged
@aabadie
Copy link
Contributor

aabadie commented Dec 17, 2019

I'm fine with the changes here so please squash. We'll have to decide later whether we accept to have tests/sys_arduino broken on arduino-uno (only). Which seems to already be the case in master anyway.
Note that this problem will be fixed by either #12973 (my preferred solution) or #12962.

@aabadie aabadie added CI: run tests If set, CI server will run tests on hardware for the labeled PR CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Dec 17, 2019
@fjmolinas fjmolinas added CI: disable test cache If set, CI will always run all tests regardless of whether they have been run successfully before CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Dec 17, 2019
@fjmolinas
Copy link
Contributor Author

The two failing tests are non related, they tend to fail. But I had indeed forgotten about the examples/

    total cpu runtime: 2d:11h:55m:02.2s
---
--- run_test job results (8 failed, 759 passed, 767 total):
    failed:
    tests/nordic_softdevice/nrf52dk:gnu
    tests/driver_nrfmin/nrf52dk:llvm
    examples/micropython/nrf52dk:gnu
    examples/micropython/samr21-xpro:gnu
    examples/suit_update/nrf52dk:gnu
    examples/suit_update/nrf52dk:llvm
    examples/suit_update/samr21-xpro:llvm
    examples/suit_update/samr21-xpro:gnu

@fjmolinas
Copy link
Contributor Author

@aabadie for now can we get away with exporting TESTRUNNER_RESET_AFTER_TERM for the examples tests? I don't think modifying examples to fit a test is the right way to go, and I want to iterate a little more over other solutions. But I would not like to stall this one anymore, it is still a big improvement from current behavior.

@aabadie
Copy link
Contributor

aabadie commented Dec 19, 2019

can we get away with exporting TESTRUNNER_RESET_AFTER_TERM for the examples tests?

+1

But I would not like to stall this one anymore, it is still a big improvement from current behavior.

I agree. With this PR we can already run the test suite on a lot more boards (remote, hifive, etc). Without it, the tests cannot be run automatically.

@fjmolinas fjmolinas removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Dec 19, 2019
@fjmolinas
Copy link
Contributor Author

@aabadie please have a look at the commit messages and the comments, I'l then trigger another build.

aabadie
aabadie reviewed Dec 19, 2019
View reviewed changes
@fjmolinas
dist/pythonlibs/testrunner: reset before term
a77594a 
For some boards `make reset` is only possible if a serial connection
is not already open or its execution might disrupt it. This
causes some tests to fail since before running a test the board
is reset.

`make reset` is currently used as a synchronization mechanism between
the application and the test script. With `test_utils_interactive_sync`
this is no longer needed so call `make reset` before `cleanterm` instead
of after when `test_utils_interactive_sync` is used.

Allow setting TESTRUNNER_RESET_AFTER_TERM=1 to keep the previous
behaviour for `examples/%/tests`.
@fjmolinas
examples: set TESTRUNNER_RESET_AFTER_TERM for examples/%/tests
ebd5acc
@fjmolinas fjmolinas added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Dec 19, 2019
aabadie
aabadie approved these changes Dec 19, 2019
View reviewed changes
Copy link
Contributor

@aabadie aabadie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK and go!

@aabadie aabadie merged commit 70e6e0a into RIOT-OS:master Dec 19, 2019
@fjmolinas fjmolinas mentioned this pull request Jan 3, 2020
Closed
@fjmolinas fjmolinas added this to the Release 2020.01 milestone Jan 15, 2020
@fjmolinas fjmolinas deleted the pr_dont_reset_sync branch July 31, 2020 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aabadie aabadie aabadie approved these changes

Assignees
No one assigned
Labels
Area: tests Area: tests and testing framework CI: disable test cache If set, CI will always run all tests regardless of whether they have been run successfully before CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation
Projects
None yet
Milestone
Release 2020.01
Development

Successfully merging this pull request may close these issues.
2 participants
@fjmolinas @aabadie