tests/sys/shell: make test script more robust #21036

maribu · 2024-11-25T10:53:09Z

Contribution description

This changes the behavior of the test script for verifying the help command: It no longer assumes a specific order for the list of commands.

Making the test robust is a bit tricky, as the module shell_cmds_default that is used here may add commands specific to a set of board. We use help_json to get the list of commands actually provided, so that we know how many rows the command table printed by help need to be parsed.

A minimum set of commands that must be present for all boards is hard-coded in the test script and the actually provided commands are tested against this. Otherwise e.g. an empty list of commands presented by help and help_json would also pass.

Testing procedure

make BOARD=samr21-xpro -C tests/sys/shell flash test

make: Entering directory '/home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell'
Building application "tests_shell" for "samr21-xpro" with CPU "samd21".

"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/pkg/cmsis/ 
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/boards/common/init
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/boards/samr21-xpro
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/core
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/core/lib
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/samd21
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/cortexm_common
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/cortexm_common/periph
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/sam0_common
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/sam0_common/periph
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/samd21/periph
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/cpu/samd21/vectors
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/drivers
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/drivers/periph_common
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/app_metadata
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/auto_init
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/div
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/frac
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/isrpipe
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/libc
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/malloc_thread_safe
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/newlib_syscalls_default
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/pm_layered
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/preprocessor
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/ps
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/shell
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/shell/cmds
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/stdio
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/stdio_uart
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/test_utils/interactive_sync
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/test_utils/print_stack_usage
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/tsrb
"make" -C /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/sys/ztimer
   text	  data	   bss	   dec	   hex	filename
  18928	   168	  2832	 21928	  55a8	/home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell/bin/samr21-xpro/tests_shell.elf
/home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/dist/tools/edbg/edbg.sh flash /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell/bin/samr21-xpro/tests_shell.bin
### Flashing Target ###
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800009225 03.25.01B6 (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Verification....
at address 0x168 expected 0xc4, read 0xc0
Error: verification failed
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800009225 03.25.01B6 (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Programming...... done.
Verification...... done.
Done flashing
/home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell/tests/01-run.py:88: SyntaxWarning: invalid escape sequence '\ '
  ("""echo "t\\e st" "\\"" '\\'' a\ b""", '"echo""te st"""""\'""a b"'),  # noqa: W605


socat - open:/dev/ttyACM0,b115200,echo=0,raw,cs8,parenb=0,cstopb=0 

> bufsize

> bufsize
60
____________________________________________________verylong
> ____________________________________________________verylong
shell: maximum line length exceeded
> garbage1234
garbage1234

> 
> shell exited

> 
> periodic 5
> 
> periodic 5
> test
test
test
test
test


> start_test
start_test
[TEST_START]
> 


> 
> echo a string
echo a string
"echo""a""string"
> echo   multiple   spaces   between   argv
echo   multiple   spaces   between   argv
"echo""multiple""spaces""between""argv"
> echo 	tabs		processed 		like			spaces
echo 	tabs		processed 		like			spaces
"echo""tabs""processed""like""spaces"
> unknown_command
unknown_command
shell: command not found: unknown_command
>      echo leading spaces
     echo leading spaces
"echo""leading""spaces"
> 					echo leading tabs
					echo leading tabs
"echo""leading""tabs"
> echo trailing spaces     
echo trailing spaces     
"echo""trailing""spaces"
> echo trailing tabs					
echo trailing tabs					
"echo""trailing""tabs"
> hello-world
hello-world
shell: command not found: hello-world                                                                                                                                   echo
echo
"echo"
> echo \'
echo \'
"echo""'"
> echo \"
echo \"
"echo""""
> echo escaped\ space
echo escaped\ space
"echo""escaped space"
> echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
"echo""escape""within""singlequote"
> echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
"echo""escape""within""doublequote"
> echo "t\e st" "\"" '\'' a\ b
echo "t\e st" "\"" '\'' a\ b
"echo""te st"""""'""a b"
> echo "hello"world
echo "hello"world
"echo""helloworld"
> echo hel"lowo"rld
echo hel"lowo"rld
"echo""helloworld"
> echo hello"world"
echo hello"world"
"echo""helloworld"
> echo quoted space " "
echo quoted space " "
"echo""quoted""space"" "
> echo abc"def'ghijk"lmn
echo abc"def'ghijk"lmn
"echo""abcdef'ghijklmn"
> echo abc'def"ghijk'lmn
echo abc'def"ghijk'lmn
"echo""abcdef"ghijklmn"
> echo "'" '"'
echo "'" '"'
"echo""'""""
> echo a\
echo a\
shell: incorrect quoting
> echo "
echo "
shell: incorrect quoting
> echo '
echo '
shell: incorrect quoting
> echo abcdef"ghijklmn
echo abcdef"ghijklmn
shell: incorrect quoting
> echo abcdef'ghijklmn
echo abcdef'ghijklmn
shell: incorrect quoting
> ps
ps
	pid | state    Q | pri 
	 1 | running  Q |   7
> xfa_test1
xfa_test1
[XFA TEST 1 OK]
> xfa_test2
xfa_test2
[XFA TEST 2 OK]
> reboot
reboot
main(): This is RIOT! (Version: 2025.01-devel-189-g3e08d6-tests/sys/shell/fix-test)
test_shell.
> end_test
end_test
[TEST_END]
> help_json
help_json
{"cmds": [{"cmd": "bufsize", "desc": "Get the shell's buffer size"}, {"cmd": "start_test", "desc": "starts a test"}, {"cmd": "end_test", "desc": "ends a test"}, {"cmd": "echo", "desc": "prints the input command"}, {"cmd": "empty", "desc": "print nothing on command"}, {"cmd": "periodic", "desc": "periodically print command"}, {"cmd": "xfa_test1", "desc": "xfa test command 1"}, {"cmd": "xfa_test2", "desc": "xfa test command 2"}, {"cmd": "app_metadata", "desc": "Returns application metadata"}, {"cmd": "pm", "desc": "interact with layered PM subsystem"}, {"cmd": "ps", "desc": "Prints information about running threads."}, {"cmd": "version", "desc": "Prints current RIOT_VERSION"}, {"cmd": "reboot", "desc": "Reboot the node"}]}
> help
help
Command              Description
---------------------------------------
bufsize              Get the shell's buffer size
start_test           starts a test
end_test             ends a test
echo                 prints the input command
empty                print nothing on command
periodic             periodically print command
xfa_test1            xfa test command 1
xfa_test2            xfa test command 2
app_metadata         Returns application metadata
pm                   interact with layered PM subsystem
ps                   Prints information about running threads.
version              Prints current RIOT_VERSION
reboot               Reboot the node
> 

> 
make: Leaving directory '/home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell'

Issues/PRs references

Fixes regression from #20958 that caused failing nightlies

This fixes: /home/marian.buschsieweke@ml-pa.loc/Repos/software/RIOT/master/tests/sys/shell/tests/01-run.py:86: SyntaxWarning: invalid escape sequence '\e' Python still assumes `\\<char>` if `\<char>` is not valid, but started to warn. Chances are good that this will be elevated to an error in future python releases.

Teufelchen1

Great! Nit: check_help() is a bit hard to read, but still fine for this test case.

maribu · 2024-11-25T12:57:27Z

check_help() is a bit hard to read, but still fine for this test case.

I sprinkled some comments over the code.

tests/sys/shell/tests/01-run.py

riot-ci · 2024-11-25T14:34:23Z

Murdock results

✔️ PASSED

b858614 tests/sys/shell: make test more robust

Success	Failures	Total	Runtime
81	0	85	01m:53s

Artifacts

Documentation preview

This changes the behavior of the test script for verifying the `help` command: It no longer assumes a specific order for the list of commands. Making the test robust is a bit tricky, as the module `shell_cmds_default` that is used here may add commands specific to a set of board. We use `help_json` to get the list of commands actually provided, so that we know how many rows the command table printed by `help` need to be parsed. A minimum set of commands that *must* be present for all boards is hard-coded in the test script and the actually provided commands are tested against this. Otherwise e.g. an empty list of commands presented by `help` and `help_json` would also pass. Co-authored-by: benpicco <benpicco@googlemail.com>

maribu · 2024-11-27T15:11:47Z

🎉 and thx :)

maribu requested review from leandrolanzieri, aabadie and MichelRottleuthner as code owners November 25, 2024 10:53

maribu added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

github-actions bot removed Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

maribu added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

maribu changed the title ~~Tests/sys/shell/fix test~~ tests/sys/shell: make test script more robust Nov 25, 2024

Teufelchen1 approved these changes Nov 25, 2024

View reviewed changes

maribu force-pushed the tests/sys/shell/fix-test branch from 3e08d6c to 4ed8197 Compare November 25, 2024 12:56

maribu force-pushed the tests/sys/shell/fix-test branch from 4ed8197 to ad4206d Compare November 25, 2024 12:58

benpicco reviewed Nov 25, 2024

View reviewed changes

tests/sys/shell/tests/01-run.py Outdated Show resolved Hide resolved

maribu force-pushed the tests/sys/shell/fix-test branch from a6c50c9 to b858614 Compare November 25, 2024 16:23

mguetschow mentioned this pull request Nov 26, 2024

boards/seeedstudio-xiao-nrf52840: initial board support #20980

Draft

Teufelchen1 added this pull request to the merge queue Nov 27, 2024

Merged via the queue into RIOT-OS:master with commit 5fa8f65 Nov 27, 2024
26 checks passed

maribu deleted the tests/sys/shell/fix-test branch November 27, 2024 15:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/sys/shell: make test script more robust #21036

tests/sys/shell: make test script more robust #21036

maribu commented Nov 25, 2024

Teufelchen1 left a comment

maribu commented Nov 25, 2024

riot-ci commented Nov 25, 2024 •

edited

Loading

maribu commented Nov 27, 2024

tests/sys/shell: make test script more robust #21036

tests/sys/shell: make test script more robust #21036

Conversation

maribu commented Nov 25, 2024

Contribution description

Testing procedure

Issues/PRs references

Teufelchen1 left a comment

Choose a reason for hiding this comment

maribu commented Nov 25, 2024

riot-ci commented Nov 25, 2024 • edited Loading

Murdock results

Artifacts

maribu commented Nov 27, 2024

riot-ci commented Nov 25, 2024 •

edited

Loading