Extremely high packet loss with UDP test

[clemensg]

Hi,

iperf2 does not report much packet loss when receiving UDP traffic on an i.MX6 quad-core processor with fec ethernet driver on Linux (4.2-rc7):

10 Mbit/s: 0% packet loss
100 Mbit/s: 0.31% packet loss

iperf3 on the other hand:

10 Mbit/s: 93% packet loss
100 Mbit/s: 99% packet loss

These values can't be right, because both iperf2 and iperf3 TCP RX tests show a maximum bandwidth of over 200 Mbit/s. It only occurs in the UDP RX test with iperf3.

This happens on iperf3.1b3 and on iperf3.0.7 and looks like a serious bug in the iperf3 UDP test. Any idea what's going on?

Cheers,
Clemens

[bmah888]

Frequently this (lossy or slow UDP tests) can be the result of using the socket buffer size. Try using the -w option to set a large socket buffer size.

[clemensg]

I did not change the socket buffer size. On the same system, with the same socket buffer size, iperf2 works fine, but iperf3 shows a huge amount of packet loss, so isn’t it more likely that there is a bug in the UDP tests of iperf3? Shouldn’t iperf3 work with default parameters at least as good/accurate as iperf2 did?

Can you reproduce this issue? If not, please try on a multi-core ARM based system, for example Raspberry Pi 2 or something similar. I was using a Freescale i.MX6Q.

On 04 Sep 2015, at 18:36, Bruce A. Mah notifications@github.com wrote:

Frequently this (lossy or slow UDP tests) can be the result of using the socket buffer size. Try using the -w option to set a large socket buffer size.

—
Reply to this email directly or view it on GitHub.

[bmah888]

I don't have any multi-core ARM systems available, and these are not platforms that we officially support..

Please try increasing the socket buffer size. I believe that iperf2 and iperf3 use different defaults.

[legraps]

Hi Bruce, I can confirm what Clemens sees. Actually it doesnt matter if I use a IMX6, OMAP or a x86 machine. Even with target bandwidth of a few MBit/s, I get packet loss of at least 3-4% up to 25%. Unfortunately setting the windows/buffer size doesnt change that (tried 500K, 1M, 2M). I can observe the behaviour even when running server and client on the same machine.
What does change the behaviour is setting the bandwidth to unlimited (-b 0). Rates go up high, e.g. 16GBit/s on a i7, and loss is 0.1%. Sort of what I expected.
As I'm otherwise well fond of using iperf3 over 2, is there anything that can be done to find the cause of the issue?
cheers,
Andreas

[grad-michal]

I observe same issue using two devices connected to switch (consumer grade, not professional) when sending device is connected through 1000BASE-T and server device is connected through 100Base-TX with bandwidth set to anything higher than 10M. Disabling autonegotiation on client side and forcing speed 100Mbit/s Full Duplex resulting in no package loss (or neglible) even with bandwidth like 90M.

Is it possible that iperf3 on fast machine sending most data at the begining of each iteration causing to buffer overflow on switch? I think iperf2 is delaying writes so that data is send more evenly in time frame.

EDIT: Sending machine has i7 onboard and I tested using both on Windows and Linux.

[wangyoucao577]

I have met an issue just like this. In the same network connection, use udp. iperf2 is ok, but iperf3 occurred a very high packet loss above 95%. Then I add an option -l in iperf3, it worked well. For iperf2, no -l option added.

[linascheryservers]

confirm. UDP test report on server side seems wrong.
Host A sending UDP stream, 1000 packets, host B (server) reports it lost 500 pakets. Testing with tcpdump/wireshark: on A sending interface packet analyzer counted 1000 pakets, on host B receiving interface packet analyzer counted 1000 packets. There is something wrong with the math inside server code.
One more thing. If host A and host B are on the same subnet - there will be no "packet loss" on iperf3 udp test. If there is any router between A and B there will be random packet loss despite tested bandwidth 20Mbps or 1Gbps.

[clemensg]

It looks like having fq_codel as qdisc does improve the situation. Can you confirm that setting tc qdisc add dev $ETH root fq_codel reduces the UDP packet loss for you too?
But in my opinion this should work with the default pfifo_fast qdisc too.. not everybody can change the queuing discipline. And it is very confusing if you get such results with default qdisc settings.

I really like the idea from @folsson (in #386) to reduce the interval of the throttling algorithm to 1ms. At the moment it is still at 100ms: https://github.com/esnet/iperf/blob/master/src/iperf_api.c#L1203

By the way, thanks to @mpitt for the IO graphs in #386! 👍

[legraps]

Cannot confirm fq_codel provides an improvement. However using plain tc qdisc add dev $ETH root fq allows to raise the bandwidth considerably (without packet loss). But only with version 3.1.3.
Also, using multiple threads (-P option) helped me to get more reliable measurements and slightly higher throuhput. I'm doing UDP Rx on an imx6solo though and at ~200MBit the CPU is loaded ~90%. What I do not understand is whyTCP is able to run about twice as fast.

[bms]

Whilst I haven't seen this issue with iperf 3, it amounts to a ground-up rewrite compared to iperf 2. The UDP code there is normally serviced by an independent thread. This can have an effect on how responsive the pacing of packet generation is; iperf 2 calculated its inter-packet times using floating point. I noticed bwping underfilling the pipe in each time window; it was using select() and integer math only to compute delays/mean packets to send. I suspect some of the underlying variables have changed in iperf 3 given the rewrite. In experimental setups with iperf 2, I've never used anything other than a first-come, first-served queueing discipline.

[bms]

Pages 197-199 of my PhD thesis https://hdl.handle.net/10023/8681 describes some of these differences. It is known that IP multipath probably requires a different approach from what iperf has now for accurate measurements, but that's really out of the scope of this specific issue. However, if the packet sourcing is bursty, that text sheds light on EXACTLY where to look. The most annoying thing about iperf 2 is how it is written in a broad C++ style, trying to treat UDP sockets like TCP ones as regards binding.

[clemensg]

@legraps Maybe TCP is able to run twice as fast because using the TCP sliding window protocol and Window Scale Option reduces RX FIFO overflows in the i.MX6 Ethernet MAC. There is an erratum (ERR004512) mentioning RX FIFO overflows. See http://cache.nxp.com/files/32bit/doc/errata/IMX6DQCE.pdf