Repository navigation
Status of running node core benchmarks #127
Description
Activity
This now seems to be working, and is ready for testing. @mcollina you mentioned you had some PRs to try it on.
At the moment it is set to test a PR with its impact on master.
There's four parameters available:
BRANCH - defaults to master
PULL_ID - the PR number
CATEGORY - the folder of benchmarks you want to run from node/benchmark
FILTER - to limit the selection of benchmarks in that folder.Please take a look and let me know how it goes!
Example of output :
https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/8/URL to launch from:
https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/build?delay=0secNOTE: Be careful to understand how long the run you are submitting will take:
Here is a guide that @AndreasMadsen wrote a year or so ago, each number (time (sec) ) that you intend to run needs multiplying by 60 (as the script will run 30 of the "old" and 30 of the "new" build.I'm currently on vacation, I'll get this going asap when I come back (1 week).
Reacted by Gareth EllisProTip: To multiple by 60 just read the seconds as minutes.
Thoughts: currently its taking AGES to clone node core, What are thoughts on cloning once on the machine and then copying over to the workspace and doing a pull origin master?
Perhaps we could update the central clone once a week?
Investigating why the run @mcollina failed - will update with findings....
How to: Running core benchmarks on Node.js CI
Current function
The current jenkins job aims at allowing the impact of a PR on a particular branch to be compared. For example how does PR XXXX impact performance on master?
How to configure the job
To start with visit https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/build?delay=0sec
Here you have four values to enter:BRANCH -
this is which branch or tag to compare to, defaults to master
PULL_ID -
pull request to test
These two variables provide the two copies of Node.js to compare - we build BRANCH, then take a copy and apply the diff of PULL_ID, and build again. We now have two copies of Node to compareCATEGORY -
This is a category of benchmarks in the node/benchmark folder from core
FILTER -
this reduces further the benchmarks in scope and is includes benchmark where FILTER exists in the filename.
On running, a minimum of 30 iterations of each benchmark/variation per build, once they have all finished Rscript will summarise the results, suggesting improvement / regression as well as confidence intervals.
Reacted by Andreas Madsen@gareth-ellis It would probably make sense to PR this as a doc somewhere in the repo.
@AndreasMadsen @mcollina @mscdex , if you have some cycles do you want to have a go with the job, using instructions above and see how you get on?
Also got a PR for the above doc with a view to getting something together we can send out to a wider audience and get people using this job and core benchmarks. (PR in #144 )
I would love to see it in action but right now I don't have anything I would like to benchmark :(
- Reacted by Andreas Madsen
I think the script should automatically rebase the branch, because a difference in the V8 engine could skew the results and provide wrong feedback. Maybe this should be a checkbox, so we can verify across different release lines.
I run the tests on my laptop in June and I got no regressions (nodejs/node#12857 (comment)) - this was at the time of V8 5.9.
However the branch is still using an extremely old V8 engine, and we got some very weird results running it now:
improvement confidence p.value net/net-c2s-cork.js dur=5 type="buf" len=1024 2.56 % 0.330368195 net/net-c2s-cork.js dur=5 type="buf" len=128 -5.73 % * 0.010873902 net/net-c2s-cork.js dur=5 type="buf" len=16 -0.34 % 0.869100857 net/net-c2s-cork.js dur=5 type="buf" len=32 3.41 % 0.218088510 net/net-c2s-cork.js dur=5 type="buf" len=4 -1.04 % 0.491132059 net/net-c2s-cork.js dur=5 type="buf" len=512 -5.45 % * 0.032530436 net/net-c2s-cork.js dur=5 type="buf" len=64 -1.61 % 0.579751020 net/net-c2s-cork.js dur=5 type="buf" len=8 -2.84 % 0.242407708 net/net-c2s.js dur=5 type="asc" len=102400 -0.86 % 0.500537810 net/net-c2s.js dur=5 type="asc" len=16777216 -0.73 % 0.762753668 net/net-c2s.js dur=5 type="buf" len=102400 0.55 % 0.664537769 net/net-c2s.js dur=5 type="buf" len=16777216 1.11 % 0.557571579 net/net-c2s.js dur=5 type="utf" len=102400 -0.39 % 0.822950308 net/net-c2s.js dur=5 type="utf" len=16777216 1.38 % 0.453838521 net/net-pipe.js dur=5 type="asc" len=102400 0.41 % 0.712092837 net/net-pipe.js dur=5 type="asc" len=16777216 0.81 % 0.775137756 net/net-pipe.js dur=5 type="buf" len=102400 0.76 % 0.466008198 net/net-pipe.js dur=5 type="buf" len=16777216 0.90 % 0.558028886 net/net-pipe.js dur=5 type="utf" len=102400 0.54 % 0.671323747 net/net-pipe.js dur=5 type="utf" len=16777216 3.82 % * 0.032465465 net/net-s2c.js dur=5 type="asc" len=102400 0.84 % 0.509765096 net/net-s2c.js dur=5 type="asc" len=16777216 -2.78 % 0.217136214 net/net-s2c.js dur=5 type="buf" len=102400 0.11 % 0.925209854 net/net-s2c.js dur=5 type="buf" len=16777216 0.03 % 0.986008241 net/net-s2c.js dur=5 type="utf" len=102400 -2.64 % 0.129138848 net/net-s2c.js dur=5 type="utf" len=16777216 2.26 % 0.088823248 net/tcp-raw-c2s.js dur=5 type="asc" len=102400 2.51 % 0.292369145 net/tcp-raw-c2s.js dur=5 type="asc" len=16777216 -0.08 % 0.972487579 net/tcp-raw-c2s.js dur=5 type="buf" len=102400 6.07 % 0.052679575 net/tcp-raw-c2s.js dur=5 type="buf" len=16777216 1.04 % 0.562458368 net/tcp-raw-c2s.js dur=5 type="utf" len=102400 5.49 % 0.217685094 net/tcp-raw-c2s.js dur=5 type="utf" len=16777216 -0.06 % 0.884432672 net/tcp-raw-pipe.js dur=5 type="asc" len=102400 0.93 % 0.731389654 net/tcp-raw-pipe.js dur=5 type="asc" len=16777216 -18.48 % 0.304436728 net/tcp-raw-pipe.js dur=5 type="buf" len=102400 -0.78 % 0.781388921 net/tcp-raw-pipe.js dur=5 type="buf" len=16777216 75.96 % 0.165276028 net/tcp-raw-pipe.js dur=5 type="utf" len=102400 -5.66 % ** 0.006928205 net/tcp-raw-pipe.js dur=5 type="utf" len=16777216 -1.05 % 0.669799761 net/tcp-raw-s2c.js dur=5 type="asc" len=102400 0.48 % 0.848273086 net/tcp-raw-s2c.js dur=5 type="asc" len=16777216 -1.09 % 0.664140660 net/tcp-raw-s2c.js dur=5 type="buf" len=102400 5.85 % * 0.046938660 net/tcp-raw-s2c.js dur=5 type="buf" len=16777216 -0.38 % 0.884173926 net/tcp-raw-s2c.js dur=5 type="utf" len=102400 -4.80 % 0.174611084 net/tcp-raw-s2c.js dur=5 type="utf" len=16777216 -0.71 % 0.364776237 Notifying upstream projects of job completion Finished: SUCCESSI kicked off the job again: https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/15/.
It would be good to access the raw data, so I can use R to plot graphs.. is that pushed somewhere?
I think the script should automatically rebase the branch, because a difference in the V8 engine could skew the results and provide wrong feedback. Maybe this should be a checkbox, so we can verify across different release lines.
The script checkouts the
$BRANCH(master) build node, then cherry-pick$PULL_IDon top and build again. So it should be rebased.However the branch is still using an extremely old V8 engine, and we got some very weird results running it now:
What are the wired results? You have to remember that there is a small risk for a false positive and with many independent test that risk becomes large. At only 5% confidence (one star), the risk of getting 5 false positive in 44 tests is 4.590962%.
plot is only valid for 44 independent test.
plot code
library(ggplot2); c1 = data.frame( false.positive=0:44, properbility=dbinom(0:44, 44, 0.05), confidence='*' ) c2 = data.frame( false.positive=0:44, properbility=dbinom(0:44, 44, 0.01), confidence='**' ) c3 = data.frame( false.positive=0:44, properbility=dbinom(0:44, 44, 0.001), confidence='***' ) c = rbind(rbind(c1, c2), c3) p = ggplot(c, aes(x=false.positive)) + geom_bar(aes(y=properbility, fill=confidence), stat="identity") + facet_grid(confidence ~ .) + scale_x_continuous(breaks=0:44, limit=c(-0.5,8))) print(p)
My bad. So, this is fine.
The script works beautifully then. I think we should document it in the main repo, it would be of great help to everyone.
3 remaining items
@mcollina , for getting to the raw data, if you click on your job, you should be able to open the workspace, and then browse to
e.g. https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/ws/benchmarking/experimental/benchmarks/community-benchmark/node/output.csv.I'll modify the script to move it to the top level, so there is less moving around. I think the gotcha here could be the workspace may be cleared before each run. I will rename the file then I can be sure. Otherwise another option may be copying it to a folder on the benchmarking machine and have it synced over to the website for download. We do something similar for the charts (at https://benchmarking.nodejs.org) so it should in theory be possible. Then we could post a link in the job output to where the file will be.
For posting to a PR, I know there's some bots - in the org - are there any i can pass data to post to a PR? In the job I already have the PR number, so I have the information to send and where it needs to go.
both approach make sense. Even uploading it to a gist on github (together with a chart, maybe), would be fantastic. Something that last for a bit of time with some meta-info (shas being tested).
Reacted by Andreas MadsenOK, will look into that.
In the meantime, csvs now get saved in the root of the ws with date/timestamp:
https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/ws/
Are we ready to announce to the rest of the team they can start using this or is there any other testing feedback we should get first.
@mhdawson I don't think there is anything to lose. It is not like making this public can cause anything else to fail or break.
@gareth-ellis is there a reason that
RUNSwasn't exposed as a parameter to the job?the main reason was that we were using the default number set of iterations set per benchmark, I can add the RUNS parameter if you like, just need to appreciate that if you set the number too low, the results may not be particularly useful, and if set too high and with enough benchmarks in scope it could take a long time to run
I looked at the code and saw that the script will use it, so I added it, because we have a PR (nodejs/node#16888 (comment)) that is very noisy, and AFAICT it would give better results with more runs but shorter iterations.
AFAICT it would give better results with more runs but shorter iterations
In theory, it will only make a tiny difference. After 30-100 runs, the difference should be less than the numerical precision. In practice, decreasing the number of iteration can change the optimization behavior.
Look at this distribution:
I think all the outliers are cases where the GC kicks in, and it's not what we're trying to look at.@refack what are the outliers?
@refack I have made a long comment in the other issue (nodejs/node#16888 (comment)), hopefully that will clarify things.
Gareth opened nodejs/node#17359 to let collaborators know about it.
Anything more to be discussed in this issue ? The benchmark job is now available, publicized and initial issues resolved.
Since the PR was closed once I'd merged in my changes, I figured I'd open this to track current progress on getting this working.
This is a continuation of #58 .
Current status is that I have the job setup, however we currently fail on the fact Rscript is missing. I have opened nodejs/build#821 to get this installed.
In the mean time I am going to create a branch in benchmarking with a change to not use Rscript, which we can then use for continued testing.