[Bug] Packet order lost when using --topspeed, --mbps or --pps with cache file

[brunkenj]

The packet order of a source pcap is lost when the options --topspeed, --mbps or --pps are used with a cache file. In most of the packet dumps that I've reviewed, it appears that tcpreplay and tcpreplay-edit play one side of the traffic first, and then the other. This does not occur when these options are used without the cache file, i.e., sending the traffic out of only one port.

test.pcap is attached for reference. NOTE: This is a traffic stream from Metasploit for CVE-2023-46604 (Apache ActiveMQ). Wireshark reports packet #4 as malformed, but none of the packets are larger than 512 bytes, and the traffic is accurately identified by Suricata.

To Reproduce

• This can be done with either a VM or a physical traffic server
• The test pcap contains only eight packets, for simplicity.
• The traffic servers used to test this issue all have three network interfaces: One for management (eth2) and the other two dedicated to the test traffic: eth0 and eth1.

The following are not required, but useful:

• The traffic ports are connected directly to each other, to prevent any anomalies external to the traffic server.
• tcpdump is run on the same traffic server as the playback, also to prevent any anomalies external to the traffic server.

1. Run tcpdump on the source pcap to view the actual packet order. Run tcpdump without -t to view the timestamps, if preferred:
   tcpdump -t -nnS -r test.pcap

2. Use tcppprep to create the cache file:
   tcpprep -a first -o test -i test.pcap

3. Open a terminal and begin a traffic capture on one of the traffic ports. All unrelated packets are filtered out in these examples:
   tcpdump -t -nnS -i eth0 not ip6 and not arp and not igmp and not port 5353

4. On the same traffic server, open a second terminal and play the pcap, using any of the mentioned replay options, with either tcpreplay or tcpreplay-edit:
   tcpreplay -i eth0 -I eth1 --topspeed -c test test.pcap

5. Review the tcpdump output to compare the original packet order with the order generated by the playback options mentioned.

Expected behavior
The packet order of the source pcap should be intact when using the higher playback rate options

Screenshots

[root@localhost admin]# tcpdump -t -nnS -i eth0 not ip6 and not arp and not igmp and not port 5353
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Actual order:

IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [S], seq 1360491770, win 64240, options [mss 1460,sackOK,TS val 530951449 ecr 0,nop,wscale 7], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [S.], seq 3913244939, ack 1360491771, win 65160, options [mss 1460,sackOK,TS val 1592861467 ecr 530951449,nop,wscale 7], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913244940, win 502, options [nop,nop,TS val 530951450 ecr 1592861467], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [P.], seq 1360491771:1360491890, ack 3913244940, win 502, options [nop,nop,TS val 530951453 ecr 1592861467], length 119
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [.], ack 1360491890, win 509, options [nop,nop,TS val 1592861471 ecr 530951453], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [P.], seq 3913244940:3913245282, ack 1360491890, win 509, options [nop,nop,TS val 1592861472 ecr 530951453], length 342
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913245282, win 501, options [nop,nop,TS val 530951454 ecr 1592861472], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [R.], seq 1360491890, ack 3913245282, win 501, options [nop,nop,TS val 530957953 ecr 1592861472], length 0

Using --topspeed --mbps or --pps:

IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [S], seq 1360491770, win 64240, options [mss 1460,sackOK,TS val 530951449 ecr 0,nop,wscale 7], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913244940, win 502, options [nop,nop,TS val 530951450 ecr 1592861467], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [S.], seq 3913244939, ack 1360491771, win 65160, options [mss 1460,sackOK,TS val 1592861467 ecr 530951449,nop,wscale 7], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [P.], seq 1360491771:1360491890, ack 3913244940, win 502, options [nop,nop,TS val 530951453 ecr 1592861467], length 119
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913245282, win 501, options [nop,nop,TS val 530951454 ecr 1592861472], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [.], ack 1360491890, win 509, options [nop,nop,TS val 1592861471 ecr 530951453], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [P.], seq 3913244940:3913245282, ack 1360491890, win 509, options [nop,nop,TS val 1592861472 ecr 530951453], length 342
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [R.], seq 1360491890, ack 3913245282, win 501, options [nop,nop,TS val 530957953 ecr 1592861472], length 0

or often:

IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [S], seq 1360491770, win 64240, options [mss 1460,sackOK,TS val 530951449 ecr 0,nop,wscale 7], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913244940, win 502, options [nop,nop,TS val 530951450 ecr 1592861467], length 0
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [P.], seq 1360491771:1360491890, ack 3913244940, win 502, options [nop,nop,TS val 530951453 ecr 1592861467], length 119
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [.], ack 3913245282, win 501, options [nop,nop,TS val 530951454 ecr 1592861472], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [S.], seq 3913244939, ack 1360491771, win 65160, options [mss 1460,sackOK,TS val 1592861467 ecr 530951449,nop,wscale 7], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [.], ack 1360491890, win 509, options [nop,nop,TS val 1592861471 ecr 530951453], length 0
IP 10.1.0.2.61616 > 10.1.0.1.38829: Flags [P.], seq 3913244940:3913245282, ack 1360491890, win 509, options [nop,nop,TS val 1592861472 ecr 530951453], length 342
IP 10.1.0.1.38829 > 10.1.0.2.61616: Flags [R.], seq 1360491890, ack 3913245282, win 501, options [nop,nop,TS val 530957953 ecr 1592861472], length 0

System

• OS: Fedora 42, CentOS 6.10
• Both OSes have all available updates installed
• Tcpreplay Versions: 4.5.1 and 4.5.2

Version info for tcpreplay on CentOS 6.10:

[root@localhost testing]# ./tcpreplay -V
tcpreplay version: 4.5.2 (build git:v4.5.2) (debug)
Copyright 2013-2025 by Fred Klassen <tcpreplay at appneta dot com> - AppNeta
Copyright 2000-2012 by Aaron Turner <aturner at synfin dot net>
The entire Tcpreplay Suite is licensed under the GPLv3
Cache file supported: 04
Compiled against libdnet: 1.12
Compiled against libpcap: 1.4.0
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Packet editing: disabled
Fragroute engine: enabled
Injection method: PF_PACKET send()
Not compiled with netmap
Not compiled with AF_XDP

[brunkenj]

test.pcap.gz

[brunkenj]

I built 4.5.2 with debug support enabled and re-ran my traffic. The debug output reports that the packets were sent in order. I've attached the debug output for reference.

debug_info.txt

[brunkenj]

One more note: I split test.pcap into two pcaps using Wireshark, exporting all of the client side packets to one pcap, and the server packets to another, and then replayed them using --dualfile. The result is the same when --topspeed is used.