[CSIT-1964] 3nb-spr, 3n-snr: Wireguardhw tests are likely to crash

[vvalderrv]

Description

This is similar to CSIT-1886, but there are notable differences. CSIT-1886 affects mainly large scale tests, this also affects small scale tests (but hw only). Some failures are similar to CSIT-1886 (NDR fails due to some tunnels reporting peer error), but some failures suggest VPP crash (which does not happen with CSIT-1886) by failing VPP PID check, although no actual core file is seen so far.

More investigation is needed to prove this is really an independent failure, not just some infrequent known issue occurring more frequently in 3nb-spr wireguardhw tests.

[0] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3nb-spr/67/log.html.gz#s1-s1-s1-s3-s5

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Fri, 15 Nov 2024 10:20:23 +0000): In another run, VPP did not crash, but stats [2] show "Peer error" on main thread and "Keypair error" on workers, so I suspect the cause is the same, just a different part of memory got corrupted.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/14/log.html.gz#s1-s1-s1-s3-s2-t6-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

• vrpolak (Fri, 15 Nov 2024 10:13:30 +0000):

  Still present in rls2410, but this time I see core [1].

Here is the relevant part:

#7

#8 0x00007fff34fd9e2a in EVP_PKEY_get_raw_public_key () from /lib/x86_64-linux-gnu/libcrypto.so.3

#9 0x00007fff2e43607b in curve25519_gen_public () at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_key.c:74

#10 0x00007fff2e4379dc in noise_create_initiation () at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_noise.c:140

#11 0x00007fff2e439d94 in wg_send_handshake (vm=vm@entry=0x7fff35e00700, peer=0x7fff3a621af8, is_retry=) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_send.c:175

#12 0x00007fff2e43f968 in wg_expired_new_handshake (vm=0x7fff35e00700, peer=) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_timer.c:187

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/13/log.html.gz#s1-s1-s1-s3-s2-t1-k3-k4-k1

Original issue: https://jira.fd.io/browse/CSIT-1964

[vvalderrv]

In another run, VPP did not crash, but stats [2] show "Peer error" on main thread and "Keypair error" on workers, so I suspect the cause is the same, just a different part of memory got corrupted.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/14/log.html.gz#s1-s1-s1-s3-s2-t6-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

[vvalderrv]

Still present in rls2410, but this time I see core [1].
Here is the relevant part:

#7 <signal handler called>
#8 0x00007fff34fd9e2a in EVP_PKEY_get_raw_public_key () from /lib/x86_64-linux-gnu/libcrypto.so.3
#9 0x00007fff2e43607b in curve25519_gen_public (<snip>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_key.c:74
#10 0x00007fff2e4379dc in noise_create_initiation (<snip>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_noise.c:140
#11 0x00007fff2e439d94 in wg_send_handshake (vm=vm@entry=0x7fff35e00700, peer=0x7fff3a621af8, is_retry=<optimized out>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_send.c:175
#12 0x00007fff2e43f968 in wg_expired_new_handshake (vm=0x7fff35e00700, peer=<optimized out>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_timer.c:187

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/13/log.html.gz#s1-s1-s1-s3-s2-t1-k3-k4-k1

[vrpolakatcisco]

Tests are still failing [3] in rls2502, although mostly without core.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2502-3n-snr/35/log.html.gz#s1-s1-s1-s3-s10-t2-k3-k4-k1

[vrpolakatcisco]

In rls2506 I found a run with core [4]:

#13 0x000055555555aa30 in os_panic () at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vpp/vnet/main.c:454
#14 0x00007ffff60eee3d in clib_mem_heap_alloc_inline (heap=0x0, size=49, align=8, os_out_of_memory_on_failure=1) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vppinfra/mem_dlmalloc.c:647
#15 clib_mem_alloc (size=49) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vppinfra/mem_dlmalloc.c:662
#16 0x00007ffff63a33d4 in vl_msg_api_alloc_internal (vlib_rp=0x13002a000, nbytes=49, pool=0, may_return_null=0) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlibapi/memory_shared.c:179
#17 0x00007ffff7f993bb in vl_api_rpc_call_main_thread_inline (force_rpc=0 '\000', fp=<optimized out>, data=<optimized out>, data_length=<optimized out>) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlibmemory/memclnt_api.c:639
#18 vl_api_rpc_call_main_thread (fp=0x7ffff5d2e140 <wg_api_peer_event_cb>, data=data@entry=0x7ffe70ffd170 <incomplete sequence \370>, data_length=data_length@entry=8) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlibmemory/memclnt_api.c:663
#19 0x00007ffff5d2e13c in wg_api_peer_event (peeri=<optimized out>, flags=<optimized out>) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/plugins/wireguard/wireguard_api.c:394
#20 0x00007ffff5cf5447 in wg_input_inline (vm=0x7fff3b097400, frame=0x7fff3b092880, is_ip4=1 '\001', async_next_node=12, node=<optimized out>) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/plugins/wireguard/wireguard_input.c:860
#21 wg4_input_node_fn (vm=0x7fff3b097400, node=0x7fff3b2499c0, frame=0x7fff3b092880) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/plugins/wireguard/wireguard_input.c:1187
#22 0x00007ffff61ab5ef in dispatch_node (vm=0x7fff3b097400, node=0x7fff3b2499c0, type=VLIB_NODE_TYPE_INTERNAL, frame=0x7fff3b092880, dispatch_reason=VLIB_NODE_DISPATCH_REASON_PENDING_FRAME, last_time_stamp=22281037440002225) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlib/main.c:938
#23 dispatch_pending_node (vm=0x7fff3b097400, pending_frame_index=pending_frame_index@entry=13, last_time_stamp=22281037440002225) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlib/main.c:1096
#24 0x00007ffff61ae53e in vlib_main_or_worker_loop (vm=0x7fff3b097400, is_main=0) at /w/workspace/vpp-merge-2506-ubuntu2404-x86_64/src/vlib/main.c:1635

Out of memory error, which might be unrelated (infra issue) or related (different random symptom due to memory layout).
Needs more investigation. It is possible that the original bug got fixed, and this last run is just a generic OOM issue, first reported as #4093 and now hitting an unrelated test.

[4] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2506-3nb-spr/12/log.html.gz#s1-s1-s1-s3-s6-t2-k3-k4-k1

[vrpolakatcisco]

Here is an example with small scale. VPP still vanishes but without [5] core.

[5] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2506-3nb-spr/13/log.html.gz#s1-s1-s1-s3-s12-t5-k3-k4-k1

[vrpolakatcisco]

A nicer stacktrace [6] from debug build:

#13 0x000055555555b253 in os_panic () at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vpp/vnet/main.c:454
#14 0x00007ffff6da4b99 in debugger () at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vppinfra/error.c:84
#15 0x00007ffff6da4950 in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7fff3638c91c "%s:%d (%s) assertion `%s' fails") at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vppinfra/error.c:143
#16 0x00007fff3621523c in cryptodev_raw_aead_enqueue_internal (vm=0x7fff382ea740, frame=0x7fff398c8940, op_type=CRYPTODEV_OP_TYPE_ENCRYPT, aad_len=0 '\000') at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/plugins/dpdk/cryptodev/cryptodev_raw_data_path.c:333
#17 0x00007fff362140f1 in cryptodev_enqueue_frame_to_qat (vm=0x7fff382ea740, ring_elt=0x7fff3722a4d0) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/plugins/dpdk/cryptodev/cryptodev_raw_data_path.c:535
#18 0x00007fff362136da in cryptodev_raw_dequeue (vm=0x7fff382ea740, nb_elts_processed=0x7ffe721ff274, enqueue_thread_idx=0x7ffe721ff272) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/plugins/dpdk/cryptodev/cryptodev_raw_data_path.c:555
#19 0x00007ffff7af67e1 in crypto_dequeue_frame (vm=0x7fff382ea740, node=0x7fff3af55480, ct=0x7fff381796c0, hdl=0x7fff362135c0 <cryptodev_raw_dequeue>, n_cache=64, n_total=0x7ffe721ff2d4) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vnet/crypto/node.c:150
#20 0x00007ffff7af631f in crypto_dispatch_node_fn_icl (vm=0x7fff382ea740, node=0x7fff3af55480, frame=0x0) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vnet/crypto/node.c:166
#21 0x00007ffff6ed1e39 in dispatch_node (vm=0x7fff382ea740, node=0x7fff3af55480, type=VLIB_NODE_TYPE_INPUT, frame=0x0, dispatch_reason=VLIB_NODE_DISPATCH_REASON_POLL, last_time_stamp=13873484191271083) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vlib/main.c:938
#22 0x00007ffff6ecd718 in vlib_main_or_worker_loop (vm=0x7fff382ea740, is_main=0) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vlib/main.c:1585
#23 0x00007ffff6ed46a6 in vlib_worker_thread_fn (arg=0x7fff36f9cf00) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vlib/main.c:2085
#24 0x00007ffff6f05756 in vlib_worker_thread_bootstrap_fn (arg=0x7fff36f9cf00) at /w/workspace/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/src/vlib/threads.c:490
#25 0x00007ffff6af7a94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#26 0x00007ffff6b84c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[6] https://logs.fd.io/vex-yul-rot-jenkins-1/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3nb-spr/27/csit_current/0/log.html.gz#s1-s1-s1-s1-s1-t1-k3-k4-k1

[vrpolakatcisco]

Still present in rls2510, mostly without [7] core.

[7] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2510-3n-emr/39/log.html.gz#s1-s1-s1-s3-s12-t4-k3-k4-k1