Crashes on route creation

[PlagueCZ]

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa49ab393ea in rte_stack_pop (s=0x0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/stack/rte_stack.h:140

warning: 140    ../lib/stack/rte_stack.h: No such file or directory
[Current thread is 1 (LWP 9)]
#0  0x00007fa49ab393ea in rte_stack_pop (s=0x0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/stack/rte_stack.h:140
#1  stack_dequeue (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../drivers/mempool/stack/rte_mempool_stack.c:59
#2  0x00007fa49d316650 in rte_mempool_ops_dequeue_bulk (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/mempool/rte_mempool.h:793
#3  rte_mempool_do_generic_get (mp=<optimized out>, obj_table=0x7fa4985dc9e8, n=<optimized out>, cache=<optimized out>) at ../lib/mempool/rte_mempool.h:1608
#4  rte_mempool_generic_get (mp=<optimized out>, obj_table=<optimized out>, n=<optimized out>, cache=<optimized out>) at ../lib/mempool/rte_mempool.h:1661
#5  rte_mempool_get_bulk (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/mempool/rte_mempool.h:1696
#6  rte_mempool_get (mp=0x11c669b5c0, obj_p=0x7fa4985dc9e8) at ../lib/mempool/rte_mempool.h:1722
#7  node_alloc (rib=rib@entry=0x11c7894f40) at ../lib/rib/rte_rib.c:91
#8  0x00007fa49d316ac5 in rte_rib_insert (rib=rib@entry=0x11c7894f40, ip=ip@entry=184156216, depth=depth@entry=32 ' ') at ../lib/rib/rte_rib.c:274
#9  0x000055b64732aa3c in dp_add_route (port=0x55b65a638490, vni=<optimized out>, t_vni=207, ip=184156216, t_ip6=t_ip6@entry=0x7fa4985dcae0, depth=<optimized out>) at ../src/dp_lpm.c:101
#10 0x000055b6472e0c4b in dp_grpc_add_route (port=<optimized out>, vni=<optimized out>, t_vni=<optimized out>, addr=0x7fa4985dcba8, t_ip6=0x7fa4985dcae0, depth=<optimized out>)
    at ../src/grpc/dp_grpc_impl.c:43
#11 dp_process_create_route (responder=0x7fa4985dcba0) at ../src/grpc/dp_grpc_impl.c:578
#12 dp_process_request (m=<optimized out>) at ../src/grpc/dp_grpc_impl.c:999
#13 0x000055b64731a8fd in handle_nongraph_queues () at ../src/nodes/rx_periodic_node.c:52
#14 rx_periodic_node_process (graph=0x11ddd14100, node=0x11ddd22f80, objs=0x11ddcd0100, nb_objs=<optimized out>) at ../src/nodes/rx_periodic_node.c:74
#15 0x000055b6473320ca in __rte_node_process (graph=0x11ddd14100, node=0x11ddd22f80) at /usr/local/include/rte_graph_worker_common.h:210
#16 rte_graph_walk_rtc (graph=<optimized out>) at /usr/local/include/rte_graph_model_rtc.h:42
#17 rte_graph_walk (graph=0x11ddd14100) at /usr/local/include/rte_graph_worker.h:38
#18 graph_main_loop (arg=<optimized out>) at ../src/dpdk_layer.c:171
#19 0x00007fa49ce4967e in eal_thread_loop (arg=<optimized out>) at ../lib/eal/common/eal_common_thread.c:199
#20 0x00007fa49ce59829 in eal_worker_thread_loop (arg=<optimized out>) at ../lib/eal/linux/eal.c:517
#21 0x00007fa49c451b7b in ?? ()
#22 0x0000000000000000 in ?? ()

We are running stress tests and the types of errors are piling up, this may even be connected to #621

It is starting to look like some memory corruption which is the simply manifesting in these "random" crashes and leads us on wild goose chases.

@FlorinPeter is implementing an AI environment that will look through dumps and run dpservice under debug.

Any fixes found this way will be linked to this issue.