test_utils: add UDP benchmark

[benpicco]

Contribution description

I needed a tool to generate configurable load on the network so I cobbled this together.
Maybe it's useful for others too.

Testing procedure

Add the benchmark_udp module to your application

USEMODULE += benchmark_udp

e.g.:

examples/gnrc_networking$ USEMODULE='benchmark_udp' make

Start the benchmark process by specifying the IP of the host running the benchmark server

benchmark start fd00:dead:beef::1

Build and start the benchmark server on your local machine

make -C dist/tools/benchmark_udp
dist/tools/benchmark_udp/bin/benchmark_server :: 12345

Observe the results:

host                       	bandwith	num RX	num TX	rate	RTT
2001:db9::d44c:5fff:fe16:1ebc	 439 b/s	111	111	100%	480 µs
2001:db9::c44:6ff:fe3b:50cf  	 474 b/s	186	186	100%	617 µs

You can use the -i <µs> option to adjust the interval in which clients will send packets.
With the -s <bytes> option you can adjust the size of the packets. E.g.

dist/tools/benchmark_udp/bin/benchmark_server :: 12345 -i 100 -s 100

to send packets with 100 byte payload every 100 µs

host                         	bandwith	num RX	num TX	rate	RTT
2001:db9::d44c:5fff:fe16:1ebc	318671 b/s	48449	55857	86%	301 µs
2001:db9::c44:6ff:fe3b:50cf  	263959 b/s	40247	42520	94%	404 µs

Issues/PRs references

[kfessel]

I did compile the server with -m32 (wanted to make sure that there is no issue with the c fixed int types and the format strings).
It compiled fine, but with -i 100 -s 100 applied to the benchmark server bw showed signs of uint overflow (printed bandwidth counting up).

maybe the average can be made sliding instead of total

✔️

[kfessel]

While testing this on native i found that this yields interesting cases for the network stack at high load 👍 good for further development

• locking up RIOT / just the network stack
  -i 100 -s 100 -> lost interrupts
  -i 300 -s 1000 -> complete unresponsiveness
  -i 300 -s 500 -> different behaviors

The Packet-Contend Format is LittleEndian (System Endianess) - I assume this is intentional, maybe this needs documentation.

The Configuration is not redone if the benchmark is stopped and started again, also the counters are not reset, since the cookie is not deleted on stop - Is this intentional?

✔️

[kfessel]

when i did multiple starts one after another the client looked up some times

RIOT network stack example application
All up, running the shell now
> benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
> benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
> benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
> benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
> benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234
benchmark start fe80::xxxx:xxxx:xxxx:4f64 1234^C
Program received signal SIGINT, Interrupt.
0x5656857a in _netreg_lookup (from=0x0, type=type@entry=GNRC_NETTYPE_UDP, demux_ctx=demux_ctx@entry=4294901760) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/netreg/gnrc_netreg.c:98
98              LL_SEARCH_SCALAR(head, res, demux_ctx, demux_ctx);
(gdb) bt
#0  0x5656857a in _netreg_lookup (from=0x0, type=type@entry=GNRC_NETTYPE_UDP, demux_ctx=demux_ctx@entry=4294901760) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/netreg/gnrc_netreg.c:98
#1  0x56568753 in gnrc_netreg_num (type=GNRC_NETTYPE_UDP, demux_ctx=4294901760) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/netreg/gnrc_netreg.c:114
#2  0x565655ee in gnrc_netapi_dispatch (type=GNRC_NETTYPE_UDP, demux_ctx=4294901760, cmd=514, pkt=0x565a1c00 <_pktbuf_buf+640>) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/netapi/gnrc_netapi.c:88
#3  0x5656d168 in gnrc_netapi_dispatch_send (pkt=0x565a1c00 <_pktbuf_buf+640>, demux_ctx=4294901760, type=GNRC_NETTYPE_UDP) at /home/k_f/ws_RIOT/RIOT_tree/sys/include/net/gnrc/netapi.h:185
#4  gnrc_sock_send (payload=0x565a1ba0 <_pktbuf_buf+544>, local=0x5659164c <send_thread_stack+7596>, remote=0x56591664 <send_thread_stack+7620>, nh=17 '\021') at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/sock/gnrc_sock.c:281
#5  0x5656da54 in sock_udp_send_aux (sock=0x565918a0 <sock>, data=0x5658d4a0 <buf_tx>, len=516, remote=0x565916c4 <send_thread_stack+7716>, aux=0x0) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/sock/udp/gnrc_sock_udp.c:375
#6  0x56558042 in sock_udp_send (remote=0x565916c4 <send_thread_stack+7716>, len=<optimised out>, data=0x5658d4a0 <buf_tx>, sock=0x565918a0 <sock>) at /home/k_f/ws_RIOT/RIOT_tree/sys/include/net/sock/udp.h:682
#7  _send_thread (ctx=0x56596574 <main_stack+11316>) at /home/k_f/ws_RIOT/RIOT_tree/sys/test_utils/benchmark_udp/benchmark_udp.c:119
#8  0xf7de2f69 in makecontext () from /lib/i386-linux-gnu/libc.so.6
#9  0x56596574 in main_stack ()
#10 0x5659d160 in __end_stack ()
#11 0x565916f8 in send_thread_stack ()
#12 0x565916fc in send_thread_stack ()
#13 0x56591700 in send_thread_stack ()
#14 0x5659d160 in __end_stack ()
#15 0x5658f8a0 in listen_thread_stack ()
#16 0x00000000 in ?? ()
(gdb) frame 5
#5  0x5656da54 in sock_udp_send_aux (sock=0x565918a0 <sock>, data=0x5658d4a0 <buf_tx>, len=516, remote=0x565916c4 <send_thread_stack+7716>, aux=0x0) at /home/k_f/ws_RIOT/RIOT_tree/sys/net/gnrc/sock/udp/gnrc_sock_udp.c:375
375         res = gnrc_sock_send(pkt, &local, rem, PROTNUM_UDP);
(gdb) p sock
$1 = (sock_udp_t *) 0x565918a0 <sock>
(gdb) p *sock
$2 = {reg = {entry = {next = 0x565918a0 <sock>, demux_ctx = 1234, type = GNRC_NETREG_TYPE_MBOX, target = {pid = 6320, mbox = 0x565918b0 <sock+16>}}, mbox = {readers = {next = 0x5658f878 <listen_thread_stack+8152>}, writers = {
        next = 0x0}, cib = {read_count = 1, write_count = 1, mask = 7}, msg_array = 0x565918c8 <sock+40>}, mbox_queue = {{sender_pid = 9, type = 33908, content = {ptr = 0xf38a0b63, value = 4085910371}}, {sender_pid = 5, type = 513, 
        content = {ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}, {sender_pid = 5, type = 513, content = {ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}, {sender_pid = 5, type = 513, content = {
          ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}, {sender_pid = 5, type = 513, content = {ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}, {sender_pid = 5, type = 33908, content = {ptr = 0xf38a0b63, 
          value = 4085910371}}, {sender_pid = 5, type = 513, content = {ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}, {sender_pid = 5, type = 513, content = {ptr = 0x565a1980 <_pktbuf_buf>, value = 1448745344}}}}, local = {
    family = 4, addr = {ipv6 = '\000' <repeats 15 times>, ipv4 = "\000\000\000", ipv4_u32 = 0}, netif = 0, port = 1234}, remote = {family = 0, addr = {ipv6 = '\000' <repeats 15 times>, ipv4 = "\000\000\000", ipv4_u32 = 0}, netif = 0, 
    port = 0}, flags = 0}
(gdb) 

maybe the listener is able to close the sock while the sender is going to write it.

The sock close might be better placed in the stop but this may cause a dangling listener_thread
✔️

[benpicco]

Ops, good catch - I totally forgot about the listener thread

[kfessel]

Is it possible to get the testing procedure you wrote up into a doc or readme.md next to either the module or server?

[kfessel]

I am ok if would like to squash all fixups untill yesterday and calm down the check-pr

[kfessel]

gnrc: working gnrc_netorting with added module specific example
lwip:does not build

examples/benchmark_udp$ LWIP=1 make
...
<RIOT>/pkg/lwip/contrib/sock/lwip_sock.c: In function ‘_addr_on_netif’:
<RIOT>/pkg/lwip/contrib/sock/lwip_sock.c:141:21: error: implicit declaration of function ‘LOCK_TCPIP_CORE’ [-Werror=implicit-function-declaration]
  141 |                     LOCK_TCPIP_CORE();
      |                     ^~~~~~~~~~~~~~~
<RIOT>/pkg/lwip/contrib/sock/lwip_sock.c:144:21: error: implicit declaration of function ‘UNLOCK_TCPIP_CORE’ [-Werror=implicit-function-declaration]
  144 |                     UNLOCK_TCPIP_CORE();
      |                     ^~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
...

is this a missing module or an error in lwip?

[benpicco]

This already got fixed in master, should I rebase?

[kfessel]

yes please squash and rebase

[kfessel]

testing BIG Packages

USEMODULE=gnrc_ipv6_ext_frag CFLAGS='-DBENCH_PAYLOAD_SIZE_MAX=2000' make

or
LWIP

[kfessel]

nice tool to check reliability and performance of networks and nodes under high pkg pressure.

[benpicco]

Thank you for the review!
Want to push that green button? 😉