Issue in cento version 2.5.251210

[aaksaro]

Hi Team,

We are continuously observing mlx stopped getting traffic on queues and cento is taking automatic restart due to kernel level OOM kill. I remember there was a bug which was causing memory leaks. Can you please help me out with this issue.

Apr 25 07:49:26 ams7nprobe03 systemd[1]: cento.service: A process of this unit has been killed by the OOM killer.
Apr 25 07:49:58 ams7nprobe03 systemd[1]: cento.service: Main process exited, code=killed, status=9/KILL
Apr 25 07:49:58 ams7nprobe03 systemd[1]: cento.service: Failed with result 'oom-kill'.
Apr 25 07:49:58 ams7nprobe03 systemd[1]: cento.service: Consumed 1month 3d 9h 6min 37.956s CPU time.
Apr 25 07:50:04 ams7nprobe03 systemd[1]: cento.service: Scheduled restart job, restart counter is at 1.
Apr 25 07:50:04 ams7nprobe03 systemd[1]: Stopped 100Gbps NetFlow v5/v9/IPFIX probe for IPv4/v6.
Apr 25 07:50:04 ams7nprobe03 systemd[1]: cento.service: Consumed 1month 3d 9h 6min 37.956s CPU time.
Apr 25 07:50:04 ams7nprobe03 systemd[1]: Starting 100Gbps NetFlow v5/v9/IPFIX probe for IPv4/v6...
Apr 25 07:50:04 ams7nprobe03 systemd[1]: Started 100Gbps NetFlow v5/v9/IPFIX probe for IPv4/v6.

root@ams7nprobe03:/etc/cento# tail -f /opt/data/nProbe/temp/traffic_rate.txt
mlx:mlx5_1@0 3.01 Gbps
mlx:mlx5_1@1 2.60 Gbps
mlx:mlx5_1@2 2.05 Gbps
mlx:mlx5_1@3 3.03 Gbps
mlx:mlx5_1@4 2.01 Gbps
mlx:mlx5_1@5 2.13 Gbps
mlx:mlx5_1@6 2.96 Gbps
mlx:mlx5_1@7 2.16 Gbps
mlx:mlx5_1@8 0.00 Gbps
mlx5_sum: 19.95
^C
root@ams7nprobe03:/

[cardigliano]

@aaksaro how much physical memory you have? Could you monitor/track the memory utilization of the cento process over time? And how long does it take to crash?

[aaksaro]

@cardigliano

This cento is installed on baremetal machine having memory 377GB. During the time of crash the memory consumption of cento was very high. I have attached the snapshot of the grap

root@ams7nprobe03:# free -h
total used free shared buff/cache available
Mem: 377Gi 47Gi 303Gi 6.0Mi 26Gi 327Gi
Swap: 63Gi 283Mi 63Gi
root@ams7nprobe03:#

[aaksaro]

@cardigliano : Currently cento is consuming 103GB

root@ams7nprobe03:~# systemctl status cento
● cento.service - 100Gbps NetFlow v5/v9/IPFIX probe for IPv4/v6
Loaded: loaded (/etc/systemd/system/cento.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2026-04-30 05:57:53 UTC; 1 day 7h ago
Main PID: 3148715 (cento-main)
Tasks: 20 (limit: 463772)
Memory: 103.3G
CPU: 1w 5d 10h 40min 17.404s
CGroup: /system.slice/cento.service
└─3148715 /usr/bin/cento /run/cento.conf

Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [PF_RINGInterface.cpp:325] Started packet capture on mlx:mlx5_1@8...
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@1
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@5
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@3
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@7
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@0
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@8
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@2
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@4
Apr 30 05:57:57 ams7nprobe03 cento[3148715]: 30/Apr/2026 05:57:57 [FlowExporter.cpp:37] Started flow exporter for mlx:mlx5_1@6
root@ams7nprobe03:#
root@ams7nprobe03:#
root@ams7nprobe03:~#

[cardigliano]

Memory utilization looks high, however it does not seem to be continuously increasing, at least from the chart I see. Please provide a chart with a longer interval. I probably need to provide you a binary able to produce a trace in case of crash, as of now I suggest to just check the output if it says Killed or Segfault (you can run it manually, in foreground, if it helps).
Please provide the output of "cat /proc/net/pf_ring/stats/*" to check how much data it is processing vs memory utilization.
Thank you.

[aaksaro]

please check the last 3 months statistics

pfring statistics.txt