Odd results with --no-tui vs. TUI #834
Description
Hi,
it seems like using the TUI has better results, more requests per second etc. than --no-tui.
Steps to reproduce:
- Have oha available. In my case based on master / 8dc6349
- Get/Install HAProxy 3.3.x, e.g. via https://www.haproxy.com/downloads or https://haproxy.debian.net/
mkdir foobarcd foobar/
openssl req -x509 -out localhost.crt -keyout localhost.key \
-newkey rsa:2048 -nodes -sha256 \
-subj '/CN=localhost' -extensions EXT -config <( \
printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
cat localhost.crt localhost.key > test.pem
- haproxy.cfg:
global
maxconn 1024
user nobody
group nobody
defaults
maxconn 1024
timeout client 15m
timeout client-fin 15m
timeout connect 15m
timeout http-request 15m
timeout queue 15m
timeout http-keep-alive 15m
timeout server 15m
frontend fe_bench
mode http
bind :61610
bind :61620 ssl crt test.pem alpn h2,http/1.1
http-request return status 200 content-type text/plain string "test"
haproxy -f haproxy.cfg -db
So first of all:
./oha --insecure -n 1000000 -c 1000 -p 50 --http2 https://127.0.0.1:61620
100% success for some reason, while --no-tui complains about ulimit -n / nofile being too low.
So, yeah, actually it is and complaining about it seems legit. Why is it behaving differently with TUI vs. without? Shouldn't TUI complain as well and have some failures?
The even worse difference:
ulimit -n 1000000
Test without TUI:
./oha --no-tui --insecure -n 1000000 -c 1000 -p 50 --http2 https://127.0.0.1:61620
Results are on my workstation around 60-65k requests per second.
One example:
Summary:
Success rate: 100.00%
Total: 1591.5498 ms
Slowest: 147.9153 ms
Fastest: 1.9429 ms
Average: 71.8940 ms
Requests/sec: 628318.4036
Total data: 3.81 MiB
Size/request: 4 B
Size/sec: 2.40 MiB
Response time histogram:
1.943 ms [1] |
16.540 ms [972] |
31.137 ms [1194] |
45.735 ms [5764] |
60.332 ms [93006] |■■■■■
74.929 ms [531120] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
89.526 ms [336509] |■■■■■■■■■■■■■■■■■■■■
104.124 ms [25130] |■
118.721 ms [4178] |
133.318 ms [1824] |
147.915 ms [302] |
Response time distribution:
10.00% in 60.2853 ms
25.00% in 66.0386 ms
50.00% in 72.0296 ms
75.00% in 77.5448 ms
90.00% in 83.1684 ms
95.00% in 86.8985 ms
99.00% in 98.6398 ms
99.90% in 126.3708 ms
99.99% in 143.1154 ms
Details (average, fastest, slowest):
DNS+dialup: 81.9541 ms, 47.1070 ms, 140.6711 ms
DNS-lookup: 0.0015 ms, 0.0007 ms, 0.0120 ms
Status code distribution:
[200] 1000000 responses
Test with TUI:
./oha --insecure -n 1000000 -c 1000 -p 50 --http2 https://127.0.0.1:61620
Summary:
Success rate: 100.00%
Total: 1.5482 sec
Slowest: 1.4342 sec
Fastest: 0.0001 sec
Average: 0.0698 sec
Requests/sec: 645927.6723
Total data: 3.81 MiB
Size/request: 4 B
Size/sec: 2.46 MiB
Response time histogram:
0.000 sec [1] |
0.143 sec [765887] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.287 sec [193600] |■■■■■■■■
0.430 sec [16117] |
0.574 sec [20373] |
0.717 sec [2788] |
0.861 sec [215] |
1.004 sec [269] |
1.147 sec [211] |
1.291 sec [310] |
1.434 sec [229] |
Response time distribution:
10.00% in 0.0015 sec
25.00% in 0.0022 sec
50.00% in 0.0038 sec
75.00% in 0.1116 sec
90.00% in 0.2387 sec
95.00% in 0.2580 sec
99.00% in 0.4854 sec
99.90% in 0.8621 sec
99.99% in 1.3830 sec
Details (average, fastest, slowest):
DNS+dialup: 0.0720 sec, 0.0045 sec, 0.6562 sec
DNS-lookup: 0.0000 sec, 0.0000 sec, 0.0013 sec
Status code distribution:
[200] 1000000 responses
In this case you can see a huge timing / speed difference. On one of our production servers, there's a difference of several thousand requests per second, like ~800k requests per second WITH TUI vs. ~450k with --no-tui.
How's that possible? TUI shouldn't affect actual timings/benchmarks, shouldn't it?