Performance tips: Bonded connections don't seem to be working elsewhere than the speed tests triggered in Speedify Web UI

[onnimonni]

Sorry for bothering you again but I have now been talking to the speedify support for few times and they seem pretty clueless.

Problem with bonded bandwidth

When you're using this setup do you get the maximum bonded speeds outside of Speedify native speedtests?

The redundant mode seems to be working well and the low speed bonding too but when I'm using higher speed (>30mbps) connections the bonding doesn't seem to actually increase my bandwidth at all even though I have set all of my connections to primary.

Speedtests from the router

I ran 2 speedtests on my GL-X3000 router itself to remove the wifi speed issues out of this question:

Speedtest using CLI

root@GL-X3000:~# speedtest-cli
Retrieving [speedtest.net](http://speedtest.net/) configuration...
Testing from Seflow S.N.C. Di Marco Brame' & C. (185.213.20.216)...
Retrieving [speedtest.net](http://speedtest.net/) server list...
Selecting best server based on ping...
Hosted by Sky Wifi (Milano) [1.44 km]: 73.795 ms
Testing download speed................................................................................
Download: 123.63 Mbit/s
Testing upload speed......................................................................................................
Upload: 25.82 Mbit/s

Speedtest using Speedify Web UI

Button to trigger the tests:

Test results:

Comparing results

What is pretty weird for me is that even when I have set all of my 3 connections to "primary" to force the bonding according to https://speedify.com/blog/combining-internet-connections/fix-slow-bonding-speeds/amp/.

Step 3: Check Speedify connection priority settings. Setting all connections to Primary means Speedify will always try to bond them.
If a connection is set to Secondary (default for cellular connections), Speedify will avoid trying to use it unless the Primary connection is fully used or not working. If the Primary connection is above 30 Mbps, a Secondary connection will not be used at all.
Backup connections will never be used for bonding.

And even if I do that I'm not able to get the over 200mbps connections anywhere else than directly in the speedify speedtests.

I can clearly see traffic going through multiple connections in the graphs. Could this be happening because of some misconfiguration on my end with the OpenWrt interfaces?

[onnimonni]

Related to this I found that the native Logs generating button doesn't seem to work:

speedify-logs-not-working.mov

I can document this caveat into Readme. The log button in the unofficial page seems to work though and contains all of the logs from the speedify service.

[TalalMash]

ssh to your router or install luci-app-ttyd from System->Software to use the web UI console, then run iperf3:

opkg update
opkg install iperf3

iperf3 -s

Run the following on your Mac line-by-line to test your router performance:

brew install iperf3

## Testing single TCP download performance over Wi-Fi to the router
iperf3 -c <routeriphere> -R -P1

## Testing multiple TCP download performance over Wi-Fi to the router
iperf3 -c <routeriphere> -R -P20

## Testing single TCP upload performance over Wi-Fi to the router
iperf3 -c <routeriphere> -P1

## Testing multiple TCP upload performance over Wi-Fi to the router
iperf3 -c <routeriphere> -P20

Running iperf3 on the router and not on a computer connected via ethernet is inaccurate for testing Wi-Fi, but this simulates a similar CPU load to running a VPN on the device, testing Wi-Fi performance under system load.

If multiple TCP is much faster, then it's an indication that the Wi-Fi is lossy only when heavily loaded, and having multiple TCP compensates for retransmissions, this is usually a router/client performance issue.

If the speed over the internet (w/o Speedify) is fast despite iperf3 readings, then hardware network acceleration can be the reason which only works with direct Wi-Fi to Ethernet connection or NAT Ethernet to Ethernet, on some routers packets completely bypass the CPU, you can disable it here and check if the speed degrades while running with Speedify off and using speedtest.net on the client/laptop:

#Router
opkg update  
opkg install stress

stress --cpu 4

If the speed doesn't drop then it's likely that the router CPU isn't the bottleneck.

Note that running the Speedify speed test is multi TCP from my tests.
Not all speed testing apps like (e.g speedtest.net) always open that many streams. Edit; more than 4

[TalalMash]

Related to this I found that the native Logs generating button doesn't seem to work:

speedify-logs-not-working.mov
I can document this caveat into Readme. The log button in the unofficial page seems to work though and contains all of the logs from the speedify service.

Yes this should be noted, may also be added as a tab in the Speedify LuCI app.
Restart Speedify in the Speedify UI doesn't work either.

[onnimonni]

Thanks again for the great answers. I'll test these out.

Not all speed testing apps like (e.g speedtest.net) open that many streams.
Are you sure?

I thought that this default option enables multiple TCP connections to speedtest servers?

[TalalMash]

Correct, the commands mentioned earlier measure the speed between your laptop and the router not the internet, it also measures the speed Wi-Fi to CPU instead of Wi-Fi to ethernet, similar to Speedify speed test button being CPU-Ethernet, instead of Wi-Fi-CPU-Ethernet which bypasses acceleration twice.

Note that those speeds (~200Mbit) are already borderline high considering that the Spitz AX is around the same speed in single core performance as the Slate AX / Flint although with 2 less cores.

Slate AX is reported to max out at 320Mbit but with very large buffer bloat. (~800ms).
You can also try switching Speedify transport mode to single TCP (TCP) then install TCP BBR congestion controller for less CPU time:

opkg update
opkg install kmod-tcp-bbr
reboot

[onnimonni]

I did finally have time to continue checking this:

I get roughly 666 Mbits/sec download and upload 323 Mbits/sec with iperf3. They only differed very minimally when using single or multiple TCP connections. This was between the GL-X3000 router and my laptop.

I tried to run the speedify perf3 speedtest with the extra stress to the system and the results were still roughly the same.

I did try to disable the network acceleration and even that didn't have any big effect.

Speedify support mentioned that I should try following changes in addition:

That said, there are three settings you can try changing around that might improve the speed on client devices. You also might need to experiment with the different settings on and off to see what performs best with your particular connections.

Please do make sure to check the performance after changing each setting to see how it affects the performance and find the best combination:

Jumbo MTU turned off: /usr/share/speedify/speedify_cli jumbo off
(default is on)

Transport to UDP: /usr/share/speedify/speedify_cli transport udp
(default is auto)

Just changing those two might be enough, but there are times when packet aggregation has an effect as well. If you are still seeing slower performance on client devices after changing the two settings above, you can also try:

Turn off packet aggregation: /usr/share/speedify/speedify_cli packetaggr off
(default is on)

I'll try them next.

[onnimonni]

And the options above didn't really help either. Only made the connections worse.

What I'm wondering is that the iperf3 between the router and speedify seems still great:

The CPU usage doesn't seem to get higher than ~40% and the highest load average I was able to see was 1.7 which should be okay with my 2 CPU cores.

I'm not sure if the software on my side is somehow configured in a stupid way but nothing except the speedify iperf3 doesn't seem to get +200Mbps connections.

I'll probably try next the public iperf3 servers mentioned here: https://iperf.fr/iperf-servers.php

[TalalMash]

Interesting, I forgot to mention some kernel tweaks applied, not sure if glinet has them added already considering they have VPN apps: /etc/sysctl.conf
Another is irqbalance