Proposal for nightly code coverage run

[CurryKitten]

We'd like to get code coverage run every night and published to codecoverage.nodejs.org

There has been some great work done already as discussed in:
#31

To produce these coverage results, we believe the steps are as follows:

• Move the patches in https://github.com/addaleax/node-core-coverage to https://github.com/nodejs/testing/coverage
• Create a jenkins job (initially for master) which builds Node.js with code coverage enabled. This job will initially use the patches from https://github.com/nodejs/testing/coverage and will build Node.js and run the core Node.js tests (test-ci target). It will be configurable so that it can be run on any particular branch. (this job takes about 15-20 minutes to run)
• Make the jenkins job push the resulting coverage html files to a directory on the local machine. We will assume the coverage job will be only run for x86 and will run on the Benchmarking machine.
• Setup an hourly job on the website infra which will pull the coverage files from the benchmarking machine and put them in the correct place for them to be viewed from coverage.nodejs.org. The files will be in a directory that matches the date of the run so that coverage can be compared across time.
• In order to avoid ongoing conflicts between the patches and the Node.js source, move as many of the changes applied by the patches into the Node.js build files themselves.
• Configure backups so that historical data is maintained and can be recovered if the benchmarking machine needs to be re-imaged. Each run seems to take about 12MB.
• Decide how many days of data we can afford to retain.
• Consider if we need results for more branches than master and add jobs for those branches as appropriate.
• Look at adding additional tests to what is run in order to get a more complete picture of our coverage (for example running CITGM in addition to the base Node.js tests)
• Add in automatic triggers to send emails to the testing team when coverage drops by more than X%
• Analyse the results and look for gaps in coverage and submit new tests to close any coverage gaps

If we agree this is the right thing to do and we should move forward I'll volunteer to start working on these steps. I've talked to @mhdawson and he can help out with the parts that require creating jenkins jobs.

FYI @addaleax, @Trott

[Trott]

@nodejs/build

[addaleax]

Hi @CurryKitten! :)

I believe @jbergstroem was already working on something in that direction?

[jbergstroem]

Sounds good! @mhdawson: you want to carry this? I was talking to @Trott and @addaleax on how to (optimally) run this as part of a pr review but it's a bit too expensive for us at the moment.

[mhdawson]

I happy to help move this forward, create CI jobs, etc and pull in other build team members when necessary for pieces like pushing the files over to the website.

[jbergstroem]

@mhdawson perhaps slap a label on the same hosts that run v8test (ish) and use those seeing how they have more cpu/ram to spare. test-rackspace-fedora24-x64-1 being a good example. We'd need a new playbook for required packages though.

[mhdawson]

@jbergstroem the reason I was going to suggest the benchmarking machine is because as I understand the website needs to pull fro a known location.

[jbergstroem]

@mhdawson sgtm as long as we can make sure there's no benchmarking going on.

[mhdawson]

Staring to integrate here:

https://ci.nodejs.org/job/node-test-commit-linux-coverage-mdawson/11/nodes=benchmark/consoleFull

Still work to do, adding bit by bit so that I understand, also modifying so that it is the same as our standard build job except for the additions required to generate the code coverage.

So far have added up to the point where it builds and runs the tests.

[mhdawson]

Ok, job is setup to run as documented here:

#37

In the interim I have data being pushed here nodejs.devrus.com:8080 (one of my machines) so that people can see what it will look like.

@addaleax could you submit a PR adding these files to github.com/nodejs/testing/coverage:

gcovr-patches.diff
generate-index-html.py
patches.diff

I'll then submit a PR for my updates to generate-index-html.py.

@jbergstroem, @rvagg can you review #37 and look at adding the pull from the benchmarking machine and serving the files as coverage.nodejs.org ? Just let me know if you need any additional information. The same key used to pull the charts for benchmarking.nodejs.org can be used to do the pull in a similar manner. The content will be at ~/coverage-out/out

[mhdawson]

As an FYI I also renamed the job to remove my name so its now: https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage/

[rvagg]

done, live @ https://coverage.nodejs.org/, config @ nodejs/build#516