Dropped packets, problems communicating with RU when using DPDK (using DPAA2)

[nicbn]

I've been having problems to connect to the RU when using DPDK. The connection succeeds without DPDK, but with it, all traffic is reported as "filtered" by the switch. I'm using a NXP board (DPAA2 driver). The DPDK testpmd worked seemingly fine.

Console output:

# DPRC=dprc.2 taskset -c 1-15 ./build/apps/gnb/gnb -c configs/gnb.yml

--== srsRAN gNB (commit a041e3162) ==--

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
fslmc: Skipping invalid device (power)
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
PMD: dpni.2: netdev created, connected to dpmac.7
TELEMETRY: No legacy callbacks, legacy socket not created
Initializing the Open Fronthaul Interface for sector#0: ul_compr=[BFP,9], dl_compr=[BFP,9], prach_compr=[BFP,9], prach_cp_enabled=true, downlink_broadcast=false
Cell pci=134, bw=40 MHz, 2T2R, dl_arfcn=655000 (n77), dl_freq=3825 MHz, dl_ssb_arfcn=654048, ul_freq=3825 MHz

N2: Connection to AMF on 192.168.10.44:38412 completed
fslmc: Thread not affined to a single core
==== gNB started ===
Type <h> to view help
fslmc: Thread not affined to a single core
^CStopping ..
Could not stop application after 5 seconds. Forcing exit.
Killed

Part of the log:

2025-02-12T18:10:40.399240 [DU-F1   ] [I] F1 Setup: Sending F1 Setup Request to CU-CP...
2025-02-12T18:10:40.399244 [DU-F1   ] [I] Tx PDU du=0 tid=0: F1SetupRequest
2025-02-12T18:10:40.400957 [CU-CP-F1] [I] Rx PDU du=0 tid=0: F1SetupRequest
2025-02-12T18:10:40.401112 [CU-CP-F1] [I] Tx PDU du=0 tid=0: F1SetupResponse
2025-02-12T18:10:40.401127 [DU-F1   ] [I] Rx PDU du=0 tid=0: F1SetupResponse
2025-02-12T18:10:40.401133 [DU-F1   ] [I] F1 Setup: Procedure completed successfully.
2025-02-12T18:10:40.436361 [DU-MNG  ] [I] DU manager started successfully.
2025-02-12T18:10:40.436364 [DU      ] [I] DU-High started successfully
2025-02-12T18:10:40.436369 [DU      ] [I] DU started successfully
2025-02-12T18:10:40.436374 [OFH     ] [I] Starting the operation of the Open Fronthaul interface
2025-02-12T18:10:40.436376 [OFH     ] [I] Starting the realtime timing worker
2025-02-12T18:10:40.436402 [OFH     ] [I] Started the realtime timing worker
2025-02-12T18:10:40.436471 [OFH     ] [I] Started the DPDK ethernet frame receiver on port '0'
2025-02-12T18:10:40.436473 [OFH     ] [I] Started the operation of the Open Fronthaul interface
2025-02-12T18:10:40.437215 [SCHED   ] [I] [  435.18] Processed slot events pci=134: Cell creation idx=0 pci=134
2025-02-12T18:10:40.438707 [PHY     ] [I] [   436.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=34.5us
2025-02-12T18:10:40.438710 [PHY     ] [I] [   436.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=35.4us
2025-02-12T18:10:40.439057 [PHY     ] [I] [   436.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=6.7us
2025-02-12T18:10:40.439059 [PHY     ] [I] [   436.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=6.8us
2025-02-12T18:10:40.439571 [PHY     ] [I] [   436.4] NZP-CSI: prb=[0, 106) row=3 Ki=10 l0=4 l1=2 CDM=fd_CDM2 density=one t=18.5us
2025-02-12T18:10:40.441343 [PHY     ] [I] [  435.19] PRACH: rsi=1 rssi=-infdB detected_preambles=[] t=27.8us
2025-02-12T18:10:40.448047 [PHY     ] [I] [     0.0] SSB: pci=134 ssb_idx=0 L_max=8 common_scs=30 sc_offset=8 offset_PointA=6 pattern=C t=496.5us
2025-02-12T18:10:40.450578 [PHY     ] [I] [  436.19] PRACH: rsi=1 rssi=-infdB detected_preambles=[] t=8.7us
2025-02-12T18:10:40.457558 [PHY     ] [I] [     0.0] SSB: pci=134 ssb_idx=0 L_max=8 common_scs=30 sc_offset=8 offset_PointA=6 pattern=C t=23.1us
2025-02-12T18:10:40.458572 [PHY     ] [I] [   438.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=11.0us
2025-02-12T18:10:40.458578 [PHY     ] [I] [   438.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=8.9us
2025-02-12T18:10:40.459058 [PHY     ] [I] [   438.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=6.4us
2025-02-12T18:10:40.459061 [PHY     ] [I] [   438.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=6.6us
2025-02-12T18:10:40.459556 [PHY     ] [I] [   438.4] NZP-CSI: prb=[0, 106) row=3 Ki=10 l0=4 l1=2 CDM=fd_CDM2 density=one t=12.0us
2025-02-12T18:10:40.460578 [PHY     ] [I] [  437.19] PRACH: rsi=1 rssi=-infdB detected_preambles=[] t=3.6us
2025-02-12T18:10:40.461955 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
(repeats multiple times)
2025-02-12T18:10:40.467554 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.467567 [PHY     ] [I] [     0.0] SSB: pci=134 ssb_idx=0 L_max=8 common_scs=30 sc_offset=8 offset_PointA=6 pattern=C t=20.6us
2025-02-12T18:10:40.467635 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
(repeats multiple times)
2025-02-12T18:10:40.470508 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.470567 [PHY     ] [I] [  438.19] PRACH: rsi=1 rssi=-infdB detected_preambles=[] t=4.1us
2025-02-12T18:10:40.470589 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
(repeats multiple times)
2025-02-12T18:10:40.477498 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.477560 [PHY     ] [I] [     0.0] SSB: pci=134 ssb_idx=0 L_max=8 common_scs=30 sc_offset=8 offset_PointA=6 pattern=C t=16.8us
2025-02-12T18:10:40.477579 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
(repeats multiple times)
2025-02-12T18:10:40.478483 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.478557 [PHY     ] [I] [   440.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=7.2us
2025-02-12T18:10:40.478560 [PHY     ] [I] [   440.2] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=7.2us
2025-02-12T18:10:40.478564 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
(repeats multiple times)
2025-02-12T18:10:40.478986 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479059 [PHY     ] [I] [   440.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=4 l1=2 CDM=no_CDM density=three t=6.2us
2025-02-12T18:10:40.479062 [PHY     ] [I] [   440.3] NZP-CSI: prb=[0, 106) row=1 Ki=0 l0=8 l1=2 CDM=no_CDM density=three t=6.7us
2025-02-12T18:10:40.479067 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479102 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.3', symbol '10'
2025-02-12T18:10:40.479104 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.3', symbol '11'
2025-02-12T18:10:40.479105 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.3', symbol '12'
2025-02-12T18:10:40.479106 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.3', symbol '13'
2025-02-12T18:10:40.479226 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479309 [OFH     ] [W] DPDK dropped '3' packets out of a total of '3' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479396 [OFH     ] [W] DPDK dropped '3' packets out of a total of '3' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479479 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479561 [PHY     ] [I] [   440.4] NZP-CSI: prb=[0, 106) row=3 Ki=10 l0=4 l1=2 CDM=fd_CDM2 density=one t=11.8us
2025-02-12T18:10:40.479561 [OFH     ] [W] DPDK dropped '2' packets out of a total of '2' in the tx burst while sending data to the NIC port '0'
2025-02-12T18:10:40.479564 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '5'
2025-02-12T18:10:40.479565 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '6'
2025-02-12T18:10:40.479567 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '7'
2025-02-12T18:10:40.479568 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '8'
2025-02-12T18:10:40.479569 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '9'
2025-02-12T18:10:40.479570 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '10'
2025-02-12T18:10:40.479571 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '11'
2025-02-12T18:10:40.479572 [OFH     ] [W] Sector #0: Detected '2' late downlink U-Plane messages in the transmitter queue for slot '439.4', symbol '12'

Switch statistics:

Rx Packets: 1034
Rx Unicast: 1034
Rx Multicast: 0
Rx Broadcast: 0

Rx Filtered: 1034

--

Tx Packets: 187
Tx Unicast: 0
Tx Multicast: 178
Tx Broadcast: 9

Seemingly VLAN and MAC configuration are ok, because it worked previously without DPDK with the same configuration.

One thing I've noticed but I don't think is necessarily related is that setting ofh affinity with DPDK causes the process to freeze. Currently I'm using the following pinning setup:

expert_execution:
  cell_affinities:
   -
    l1_dl_cpus: 1,2
    l1_ul_cpus: 3,4
    ru_cpus: 5,6,7,8
    l2_cell_cpus: 9,10
  affinities:
    low_priority_cpus: 12,14
    # ofh: 1-14
    ru_timing_cpu: 11

  threads:
    upper_phy:
      pdsch_processor_type: concurrent        # Optional TEXT (auto). Sets the PDSCH processor type. Supported: [auto, generic, concurrent, lite].
      nof_pusch_decoder_threads: 4            # Optional UINT (1). Sets the number of threads used to encode PUSCH.
      nof_ul_threads: 2                       # Optional UINT (1). Sets the number of upprt PHY threads to proccess uplink.
      nof_dl_threads: 2                 # Optional UINT (1). Sets the number of upprt PHY threads to proccess downlink.
    lower_phy:
      execution_profile: quad                 # Optional TEXT (quad). Sets the lower physical layer executor profile. Supported: [single, dual, quad].

There are 15 1 GB hugepages available.

Thanks

[nicbn]

I was able to successfully get data to the RU by changing the EAL arguments to include a single lcore:

eal_args: "--lcores 0@(1-15) --main-lcore 0 -n 1"

And then undoing all pinning related configuration.

However the resulting performance is bad, I'm now trying to figure out how to use CPU pinning with a single lcore (may not be possible) or add more lcores.

[RafalArciszewski]

Hi @nicbn,
I am trying to run DPDK (DPAA2) on NXP LX2160A with no luck so far. Could you share your gnb.yml config?

Regards,
Rafał ARciszewski

[nicbn]

Sorry, I'm no longer working with LX2160A. I don't have access to the file.

[Burak111]

Hi @nicbn

How did you manage the synchronization between DU and RU? DPDK makes the NIC card invisible to the ptp4l service.