Description
When benchmarking, It is very important to see the CPU usage, for find the bottlenecks.
With this benchmark using 3s per fw, it is not easy to control the CPU usage.
But changing to 30s per fw, It's possible.
The slowest frameworks are saturating the CPU (~100% usage), and that is correct. But the fastest frameworks only ~70% CPU usage. And the CPU usage increase with the low performant frameworks.
The problem is ab that can't give enough food (requests) to the fastest frameworks. We are benchmarking the ab performance, not the php performance.
Core i7 860 @ 2.80GHz × 8 PHP 5.5.9
ab -c 10 -t 3 http://localhost/hello.php(.html)
wrk -t10 -c10 -d3s --latency http://localhost/hello.php(.html)
| ab | requests per second | transfer/second Mb | CPU usage % |
|---|---|---|---|
| hello.php | 13,373.38 | 2,80 | 40-45 |
| hello.html | 14,254.07 | 3,48 | 35-40 |
| wrk | requests per second | transfer/second Mb | CPU usage % |
|---|---|---|---|
| hello.php | 35,285.86 | 6.77 | ~90 |
| hello.html | 49,283.88 | 11.15 | ~90 |
But perhaps only happens with fast CPUs, so I tried with:
Core 2 Duo 2.53Ghz PHP 5.6.21
ab -c 10 -t 3 http://localhost/hello.php
wrk -t10 -c10 -d3s --latency http://localhost/hello.php
| hello.php | requests per second | transfer/second Mb | CPU usage % |
|---|---|---|---|
| ab | 2,060.37 | 0.51 | ~40 |
| wrk | 5,621.32 | 1.30 | ~95 |
Exactly the same, we calculate the ratio:
| CPU model | wrk | / | ab | Ratio |
|---|---|---|---|---|
| Duo | 5,621.32 | / | 2,060.37 | 2.72831 |
| i7 | 35,285.86 | / | 13,373.38 | 2.63851 |
The ratio is very similar, so It's only an ab performance limitation, independent of which CPU.
This ratio decrease proportionally to the low performance of the framework. Where ab is fast enough for the slowest frameworks.
For example, with the core i7, one of the fastest fw go from ~10.000 to ~16.000 req/s (more than hello.html with ab). And one of the slowest fw go only from 603 to 634 req/s.
This actual benchmark, is completely limited by ab performance.
Please use wrk, a modern HTTP benchmarking tool.