Benchmark is misconceiving

[Allendar]

I've been using all 3 of these languages a lot over the last few years (PHP5 and PHP7 as for PHP). These benchmarks suffer based on the different implementations the Redis have in the different languages.

Good benchmarks for these kind of broader comparison should compare multiple situations that look a lot more like real-world examples.

My proposal would be to redo this and then actually do:

• base calls with no content
• simulate base API calls (without any external libraries that might affect the language, but only core modules)
• simulate longer-running calls (also based on core modules only)

I'll try to enrich this post later with some real benchmarks, but I have to disappoint you that when you actually do these runs validly, on Go v1.6 (and 1.5 too) Node.JS and PHP will shit their pants by far.

Saying this is PHP's performance is wrong, it's benefitting on an external tool (Nginx/CGIfast/Apache). And added to that saying that caching is enabled is even more cheating. Do you know what happens with Go when you cache native objects and do live recompiles with no down-time? You get nanosecond-results. PHP can't even stay under hundreds of microseconds just by starting up. Even with (semi-implemented) JIT in PHP7 you can't achieve speeds in PHP that would match any compiled implementation.

Finally you're suffocating both Go and NodeJS' performance by using ab. ApacheBench can't handle tens of thousands of requests per second; it will drown in the NodeJS/Go performance of sequential request handling (mostly due filesystem limitations). Use wrk, sniper or vegeta to actually see what the systems are capable of.

I just reread my post and it might come up a bit harsh, but I'm merely trying to portray that what this repo tries to show is misconceiving. PHP7 on it's own is hundreds (sometimes thousands of times) slower than Go. Even behind an optimized server (preferably Nginx) it can't outrun Go. I have a Go API running with a native in-memory caching (without library overhead) and doing requests that easily get responses within a few microseconds (thousands of nanoseconds). And that's even over HTTPS. This is not bragging about my code, but merely showing that PHP is still much slower. Not to forget that if you would use something like Laravel, Symphony, CodeIgniter or ZEND you would have such immense degradation in performance. Also with a library like endless you can make your Go-installation indestructible, guaranteeing no down-time, even on DDoS. We did a DDoS test some weeks ago on a highly optimized Nginx PHP7 installation and it took about 3 seconds and the whole server have a server-load over 150.0 (CentOS) and fully crashed. Go's HTTP package is graceful and won't be daunted by overhead. It will deliver much higher loads of successful responses (Nginx, even on ab returns so many fails over NodeJS and Go). Running endless under Google App Engine with Google's package to auto-upgrade under heavier load can make a Go API written in a few days deliver millions of requests per second. From my experience working in PHP-based companies, for very big customers, you need semi-rocket-scientist that tweak your Nginx/Apache and Linux distro constantly to upkeep all the issues that come by in production. NodeJS and Go are just better and more modern solutions in so many more ways than just performance. Both will save you money, a LOT, trust me :P.

[borislemke]

@Allendar Check this out. I made my own test on my development MacBook.
Node.js is waaaay faster than PHP
https://github.com/borislemke/nodejs_vs_php

[Allendar]

@borislemke That's some amazing work! I will keep this thread open and try to see if I can write a similar comparison test with Go 1.7 somewhere this week.

Could you maybe add HTTPS benchmarks to your tests too? It's amazing how different languages deal with TLS, especially over HTTP. I'm really curious what the differences will be on Node.js and PHP. Also; the failure rates on Apache/Nginx (can't see them in your tests) will be pretty high under high loads. I'll show in the Go benchmarks that sending 20k requests will return 20k succes-responses, where PHP under Apache will fail hundreds, sometimes thousands, of those requests.

[Allendar]

Just to give you a taste of Go 1.7 otherworldly raw performance over both Node.js and PHP:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Print("Hello world")
    })
    http.ListenAndServe(":8080", nil)
}

$ wrk -d5s -t10 -c5000 http://localhost:8080
Running 5s test @ http://localhost:8080
  10 threads and 5000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.06ms    0.88ms 157.71ms   75.40%
    Req/Sec     5.69k     3.00k   12.14k    58.00%
  198228 requests in 5.08s, 21.93MB read
  Socket errors: connect 4759, read 84, write 0, timeout 0
Requests/sec:  38996.96
Transfer/sec:      4.31MB

53,525% faster.

Some runs wrk can't even process it (unable to create thread 9: Too many open files).

Update
I'm running a 15'inch MacBook Pro 2015 Retina with SSD tho. Maybe that made the difference. Maybe you can test this code on your laptop too.

[Allendar]

The formatter even made the performance suffer a bit. Sending output straight to the response writer is even faster:

package main

import "net/http"

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello world"))
    })
    http.ListenAndServe(":8080", nil)
}

$ wrk -d5s -t10 -c5000 http://localhost:8080
Running 5s test @ http://localhost:8080
  10 threads and 5000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.99ms    1.05ms 156.05ms   89.01%
    Req/Sec     6.05k     4.37k   46.89k    74.56%
  241474 requests in 5.10s, 29.48MB read
  Socket errors: connect 4759, read 92, write 0, timeout 0
Requests/sec:  47315.60
Transfer/sec:      5.78MB

[Allendar]

Was still running Xcode, Safari, Chrome etc and had external screen attached. When running the test with nothing open and hardware unattached wrk won't even run, haha:

$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files

Lowering the requests to 2500 kind of gives the same stable results as the previous 5000:

$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
  10 threads and 2500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.14ms  678.02us  11.98ms   46.16%
    Req/Sec     5.23k     2.80k   11.19k    63.54%
  238319 requests in 5.10s, 29.09MB read
  Socket errors: connect 2259, read 92, write 0, timeout 0
Requests/sec:  46701.91
Transfer/sec:      5.70MB

[Allendar]

With the IRIS net/http replacement-library (as the default HTTP library for Go is progressing slower, due to standards and safety etc.) it goes even faster:

package main

import "github.com/kataras/iris"

func main() {
    m := iris.New(iris.Configuration{
        IsDevelopment: false,
        DisableBanner: true,
        Gzip:          true,
    })

    m.Get("", func(c *iris.Context) {
        c.Write("Hello world")
    })

    m.Listen(":8080")
}

$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
  10 threads and 2500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.92ms  536.50us  14.73ms   81.86%
    Req/Sec     6.17k     4.28k   13.62k    57.68%
  311793 requests in 5.10s, 42.22MB read
  Socket errors: connect 2259, read 96, write 0, timeout 0
Requests/sec:  61082.81
Transfer/sec:      8.27MB

[Allendar]

Gotta admit on the primes, Node.js seems faster. Not sure if it's my rebuild of the getPrimes call tho or that the C-wrapper in Node.js is faster with maps than Go in this case. Also Go's encode/json library is kind of crappy. Could retry later with ffjson.

Update

Turns out I was looking at the wrong table. Seems to be a bit faster than Node.js (14,290.14 req/s).

package main

import "github.com/kataras/iris"

func main() {
    m := iris.New(iris.Configuration{
        IsDevelopment: false,
        DisableBanner: true,
        Gzip:          true,
    })

    m.Get("", func(c *iris.Context) {
        c.JSON(200, getPrimes(1000))
    })

    m.Listen(":8080")
}

func getPrimes(max int) []int {
    var i, j int
    sieve := map[int]bool{}
    primes := []int{}
    for i = 2; i <= max; i++ {
        if ok, _ := sieve[i]; !ok {
            primes = append(primes, i)
            for j = i << 1; j <= max; j += i {
                sieve[j] = true
            }
        }
    }
    return primes
}

$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
  10 threads and 2500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.30ms   13.93ms 157.14ms   68.41%
    Req/Sec     2.12k     1.13k   13.62k    71.23%
  73933 requests in 5.10s, 55.14MB read
  Socket errors: connect 2259, read 90, write 0, timeout 0
Requests/sec:  14494.08
Transfer/sec:     10.81MB

[Allendar]

Here are the final results. Node.js sure is fast on JSON marshaling. Not sure if it's the JavaScript v8 engine of native C wrapper that makes JSON handling so fast. Go can certainly use some work on that; you can obviously see the performance degrading over time in the encoding/json library. I think there's an issue with the read-buffers that give too much overhead.

Test Condition #1
wrk -d5s -t10 -c5000 http://localhost:8080

Simple GET request that returns "Hello World"
/**
 * Test results:
 * | Rank  | Subject                      | Requests/sec  | Data/sec   | Avg. Response |
 * -------------------------------------------------------------------------------------
 * |   1   | Pure Go(Multi Thread)        | 60,480.45     | 8.19MB     | 3.96ms        |
 * |   2   | Pure Node.js(Multi Thread)   | 20,873.34     | 3.09MB     | 11.40ms       |
 * |   3   | Pure Node.js(Single Thread)  | 10,375.05     | 1.53MB     | 22.70ms       |
 * |   2   | Express(JS, Multi Thread)    | 3,426.94      | 779.76KB   | 68.92ms       |
 * |   5   | Pure PHP(Multi Thread)       | 3,147.95      | 667.17KB   | 23.28ms       |
 * |   6   | Express(JS, Single Thread)   | 2,386.69      | 543.06KB   | 97.97ms       |
 * |   7   | Laravel(PHP, Multi Thread)   | 26.23         | 101.44KB   | 392.64ms      |
 * -------------------------------------------------------------------------------------
 */

Simple GET request that returns prime numbers between 0 and 1000
/**
 * Test results:
 * | Rank  | Subject                      | Requests/sec  | Data/sec   | Avg. Response |
 * -------------------------------------------------------------------------------------
 * |   1   | Pure Go(Multi Thread)        | 14,494.08     | 10.81MB    | 16.30ms       |
 * |   2   | Pure Node.js(Multi Thread)   | 14,290.14     | 10.75MB    | 16.57ms       |
 * |   3   | Pure Node.js(Single Thread)  | 7,902.13      | 5.95MB     | 19.13ms       |
 * |   4   | Express(JS, Multi Thread)    | 3,534.16      | 2.95MB     | 66.85ms       |
 * |   5   | Pure PHP(Multi Thread)       | 3,189.39      | 1.43MB     | 29.21ms       |
 * |   6   | Express(JS, Single Thread)   | 2,206.12      | 1.84MB     | 106.68ms      |
 * |   7   | Laravel(PHP, Multi Thread)   | 44.67         | 60.28KB    | 370.60ms      |
 * -------------------------------------------------------------------------------------
 */

Test Condition #2
wrk -d30s -t10 -c5000 http://localhost:8080

Simple GET request that returns "Hello World"
/**
 * Test results:
 * | Rank  | Subject                      | Requests/sec  | Data/sec   | Avg. Response |
 * -------------------------------------------------------------------------------------
 * |   1   | Pure Go(Multi Thread)        | 58,458.82     | 7.92MB     | 4.11ms        |
 * |   2   | Pure Node.js(Multi Thread)   | 24,559.01     | 3.63MB     | 9.76ms        |
 * |   3   | Pure Node.js(Single Thread)  | 11,629.34     | 1.72MB     | 20.63ms       |
 * |   4   | Express(JS, Multi Thread)    | 4,429.92      | 0.98MB     | 54.23ms       |
 * |   5   | Express(JS, Single Thread)   | 2,689.57      | 611.98KB   | 89.16ms       |
 * |   6   | Pure PHP(Multi Thread)       | 549.22        | 116.91KB   | 27.73ms       |
 * |   7   | Laravel(PHP, Multi Thread)   | 35.35         | 226.28KB   | 144.51ms      |
 * -------------------------------------------------------------------------------------
 */

Simple GET request that returns prime numbers between 0 and 1000
/**
 * Test results:
 * | Rank  | Subject                      | Requests/sec  | Data/sec   | Avg. Response |
 * -------------------------------------------------------------------------------------
 * |   1   | Pure Node.js(Multi Thread)   | 16,802.80     | 12.64MB    | 14.30ms       |
 * |   2   | Pure Go(Multi Thread)        | 14,011.49     | 10.37MB    | 17.14ms       |
 * |   3   | Pure Node.js(Single Thread)  | 8,727.14      | 6.57MB     | 26.92ms       |
 * |   4   | Express(JS, Multi Thread)    | 4,240.04      | 3.54MB     | 56.68ms       |
 * |   5   | Express(JS, Single Thread)   | 2,404.82      | 2.01MB     | 99.70ms       |
 * |   6   | Pure PHP(Multi Thread)       | 550.48        | 251.90KB   | 27.81ms       |
 * |   7   | Laravel(PHP, Multi Thread)   | 33.48         | 216.63KB   | 417.37ms      |
 * -------------------------------------------------------------------------------------
 */

[mreschke]

Hopefully at some point this week I will re-write the entire benchmark. I too need more scenarios anyway. I will build 2 linux VMs on a single machine so no other VM's are running. One VM will run wrk or similar and access the other VM across the local 1gb network (NOT localhost). This removes wrk fighting against the other processes on the server and I want network latency in there too since most of my scenarios are over a 1gb network. I will also test actual DB calls with various databases and libraries. Thanks @Allendar and @borislemke for the info.

[Allendar]

I'm looking forward to the tests. On a "raw" perspective it's not of much use tho. Go has surpassed most (native)C/JAVA/Rust benchmarks since 1.6 and in 1.7 they added even faster garbage collection and more efficient assembly resolvers/logic. The only reason Go is knee jerking in both the tests borislemke has made is due to the fact that native net/http and encoding/json is just not that amazing in the core package. There are enough tools like fasthttp and fastjson that can solve this tho. But if you look at the basic (say raw) Hello world calls Go's many times faster than any other language.

This is all web-based performance tho. If you would literally take raw PHP vs raw Go and do some random tests with arrays, maps, heavier calculations and moving data around PHP will drown. PHP is constantly leaking memory and is very at giving it back (I think this is because the whole idea of the language is to handle short requests due to it's web-based nature). A simple loop like this:

$arr = [];
$z = 1;

for ($i = 0; $i < 1e9; $i++) {
    $arr[$i] = $z * $i;
}

takes around 123.93 seconds on my MacBook and at the end of the script it has leaked over 2 gigabytes of memory and virtual memory overload. Go only does 3.68 seconds (33,67x faster) over this process and peaks only around 78MB of memory (probably even less due to it's virtual reservation, that is actually not claimed by Go itself yet). PHP has to constantly discover what it's dealing with due to it's dynamic typing (try making $z '1', it's even worse), making it very CPU and memory intensive. It also causes massive memory fragmentation, making the maximum amount of memory that needs to be allocated immense, unnecessarily because most of that memory is just massive empty holes of reserved memory blocks.

[mreschke]

@Allendar agreed. I did not see any go tests in @borislemke post but I will test a few packages when I get there.

I am mostly interested in practical usage, not raw performance. Which is why it will be over a 1gb network, with a hard coded array converted to json, a redis query converted to json and a mysql query converted to json etc... I just need reasonably accurate results for a real-world simple server setup that I would actually use. I use these results to determine which API platform I care to build for any particular project. I know my first tests were garbage, so I am looking forward to slapping 2 physical servers on a table and getting some real benchmarks over a LAN.

I know PHP is slower and isn't even in the same ballpark, but we have a million lines of code written in it, so gotta keep it around. We have many APIs written in Laravel's Lumen, even though slow, it's our familiar language. The reason I built this benchmark is to convince myself to build certain critical APIs in something else, less familiar. If anything, it will most likely be nodejs simply becuase we write javascript almost every where now, and its blazing fast. But we'll see once I get it on the LAN with actual DB calls etc...

[Allendar]

I understand what you mean. Especially from my older PHP projects (still have some running with external parties too) it's also a good argument for finding employees and having easier to-boot knowledge.

Please try to be cautious (as I noted in my first message in this post), that implementations of Redis, MySQL etc. are very different in all 3 languages. All languages combine a lot of C-wrappers and partially implement fully native code. Go's mysql package (most popular, but not official) is almost native, using only a few C-wrappers. PHP's MySQL implementation is almost all C-wrappers. In a way you're kind of discouraging PHP's outcome in the benchmarks that way too. But again you make a point; if you just want practically comparison (I'm not sure if it can still be called a benchmark then tho) would be nice to see.

Please feel free to post in here when you're working on it (not per se when finished) and maybe we can look at things together and see what makes the comparisons most fair and reasonable.

[mreschke]

Ill be sure not to mention its a "raw benchmark" but more of an API server comparison, which is the point. Since its not a raw performance benchmark, if MySQL in PHP is slower, then it's slower. MySQL is usually part of an API in any real-world app I make, so it will be an accurate test as far as my needs are concerned. I'll be sure to set proper expectations of the entire post. Ill keep you updated. My Go is poor so I may need help on a few lines of code.

[borislemke]

@mreschke @Allendar Hey guys, I updated my benchmark to include Go + Iris. I separated vanilla Go and Iris to have a more detailed overview. Go wins the Hello world hands down but loses significant performance over Node.js at the prime numbers + json encoding(using intel-go/fastjson).

[Allendar]

@borislemke Yeah, like stated, the JSON encoding library isn't that great in Go. I think ffjson is faster than fastjson tho. It fixes a lot of overhead that native encoding/json does in Go. The primes function itself (not returning as JSON), running in a normal benchmark is way faster tho. But like @mreschke said; we're focussing on web-performance here.

To make the tests more believable we might try to make more cases of rendering/loading/stress to actually see the overall performance differences. We already know that "raw" the ranking is Go>Node.js>PHP. To get a real impression on overal use of the 3 languages for web-based performance, we need a broader spectrum (incl. Redis, MySQL, Cookie-handling, Session-handling, TLS, etc.).