AcmeAir benchmark seems very noisy

[MSLaguana]

I've been trying to benchmark Node-ChakraCore using the AcmeAir benchmark, and I'm getting a lot of noise in the throughput results. Is there any guidance on how to set up a machine for benchmarking, or what kind of environment to use to limit noise?

For my test I've been using a 4 core Ubuntu 16.04 machine with X disabled, running for 4 minutes and taking the average throughput of the last 2 minutes as my benchmark figure. I've been seeing up to 10% variance between runs on occasion. Do you also see similar noise, and what do you do to mitigate it?

[gareth-ellis]

Two things I'd suggest:

Monitor the per second throughput from jmeter. There's a switch you can pass in to the command to get it to output per second or per 5 seconds. This is useful for monitoring whether the entire run is 10% lower/higher or (more likely) there's occasional drops in throughput due to either optimisations, GC or even locking. Once you can identify if this is happening I'd enable more verbose logging from the runtime to track what is going on.

Second thing to look at is what happens CPU wise when you have these slower/faster runs? How is the CPU generally? It is a good idea to pin the chakra core process to a subset of the CPU resource, and make sure you can saturate the CPU available to chakra. Then you can also monitor what is going on with your database/jmeter. Jmeter runs in java and therefore also needs time to warmup.

Does chakra/node run with acmeair from this repo, or have you had to make other changes? I'd be interested in taking a look of possible.

[kunalspathak]

@gareth-ellis , we use the acmeair from this repo without any modifications. By the way, do you know the configuration of benchmarking machine that runs official acmeair through CI?

[gareth-ellis]

Hello, I tried to get the exact commands we run from the ci, but can't spot where abouts it is. It could be a permission thing.

But basically we run the https://github.com/nodejs/benchmarking/blob/master/experimental/benchmarks/acmeair/run_acmeair.sh

This will either use environment variable $NODE or if that's not set the node that is on the path.

What you need to try and make sure is that node uses all the resource it is given (so limit the cores/threads it's given so the remaining resource can be used by mongo and jmeter without them being bottlenecks).

Is that what you wanted to know? Or was it the specs of the machine?

For testing myself, if I can get a Ubuntu 16.04 machine up, can I just check chakra out and run?

[kunalspathak]

can I just check chakra out and run?

Yes, you can get one of the nightly version of node-chakracore and run.

[gareth-ellis]

I had a go at getting a machine going with 16.04, but I think its too old to be much use (the machine), the nightly builds didn't contain any linux builds, so I had a go at building myself, but ended up when I tried to run it getting an illegal instruction exception on node -v the machine itself is probably getting on 10 years old, but is all I can get my hands on right now.

If when you run it again, you can monitor cpu usage of the different processes.
The script itself should do that - https://github.com/nodejs/benchmarking/blob/master/experimental/benchmarks/acmeair/run_acmeair.sh#L255-L261

If you're running outside of the script, you could put together a short script and launch at the same time as you drive load.

[mhdawson]

@MSLaguana are you still looking at this ?