Install Linux perf on some Linux machines

[mmarchini]

Linux perf has been broken on V8 since the Turbofan/Ignition pipeline became the default compiler. Recently on V8 6.7, we got it back to work (through a flag). Since there are some Node.js tools and some huge Node.js deployments relying on Linux perf (and other external profilers), having tests will help to keep these tools more stable.

For those tests to work, we need Linux perf available on at least some Linux machines. We could start on Ubuntu 16.04 machines and if we want we can install it on other machines later. The package can be installed with: apt install linux-tools-generic. Is it feasible?

Ref: nodejs/node#20783

[mhdawson]

@rvagg would it make sense to have it installed in the images supporting the containerized builds ?

@mmarchini which of these two is it:

• the tests are part of the core Node.js tests but detect if perf is available and don't run if its not
• the tests are separate and we'll have a different job to run them?

[mmarchini]

The first one (if the PR lands)

[rvagg]

Are we talking about the broken --perf-basic-prof stuff here? What would tests look like for this? Is it a simple matter of passing the output through perf and asserting on the results?

I don't think we're going to be able to jam this into the containers unfortunately, even though that's a great place for these kinds of tests. perf is tied to the kernel version and that's detached from the OS once you're using containers since you use the host kernel version. Unless we have a match of host OS and container OS. Right now we do have a match, 16.04 on host and in container, but they are not updated at the same time so they get out of sync a bit (I'm not sure minor kernel versions matter much for perf, however). But I'm not sure we want to be restricted to always running the same host OS and container OS for these "sharedlibs" containers.

A better approach might just be installing perf on one of our Ubuntu LTS hosts, maybe 18.04, and let the tests detect its presence and run if its there and skip if it's not. We can lock this in to our ansible scripts so we always get linux-tools-generic installed on some of those hosts.

[bnoordhuis]

perf is tied to the kernel version and that's detached from the OS once you're using containers since you use the host kernel version.

That's the theory but in practice you can just mix them. For example: I ran stock perf 3.13 on Ubuntu 14.04 for a while against the latest, continually upgraded mainline kernel and it worked just fine.

(Yes, I run mainline kernels. I even test -rc's.)

[joyeecheung]

I usually just clone the Linux master and cd into the perf source to compile it when I need perf on my servers. Does not seem to cause problems when I upgrade kernels from time to time.

[joyeecheung]

Can we use the benchmark CI machine for this? It would be nice to have perf there, I am thinking about using perf stat in a core benchmark to measure the start up time for our lazy-loading effort.

[mmarchini]

Are we talking about the broken --perf-basic-prof stuff here?

Yes. It will be unbroken once V8 6.7 lands and the tests would help us to know in advance if changes on V8 break it again.

Is it a simple matter of passing the output through perf and asserting on the results?

Yes. The proposed test can be seen here: nodejs/node#20783

let the tests detect its presence and run if its there and skip if it's not

The proposed test already does that.

A better approach might just be installing perf on one of our Ubuntu LTS hosts, maybe 18.04

+1 for that, I don't think we need to run this test on containers (at least not for now)

Can we use the benchmark CI machine for this? It would be nice to have perf there, I am thinking about using perf stat in a core benchmark to measure the start up time for our lazy-loading effort.

Looks like a good idea, as long as we have perf on at least one machine that runs tests for nodejs/node-v8 as well.

[mhdawson]

Installing on the benchmarking machine sounds good to me (although I think it would be the 2 additional benchmark machines as opposed to the ones used for the nightly runs). The only challenge is that if we only have it installed on a single machine then we'd want to run the test nightly as opposed to being part of every regression test (we not want, but probably only be able to). If nightly is ok, then we can easily setup a job to run once a day pinned to the benchmarking machine, just like the benchmark jobs.

Ideally, we'd really like it to be run on every PR that upgrade V8 as well. If we had the nightly job then it would just be a matter of ensuring that was added to the list of what to run to validate when updating V8.

[rvagg]

let me get linux-tools-generic on some ubuntu machines we already have in CI, the test that's in the WIP should pick it up and skip otherwise so I think that'll solve this

perf on the benchmarking machines would be a good idea regardless, I'll look at that at the same time, the intel/nearform ones mostly go through ansible so I think we can just put the changes in there.

[mhdawson]

@rvagg getting it on some of the ubuntu machines is good, but I think we want to be sure the tests runs regularly and I don't think we can be confident we'll get that regularly through chance.

Just to say I don't think we only want to rely on the chance that it runs on those machines during the regular regression runs. Once it is installed on some of the machines we can set up a job that only runs on that subset and make sure it runs at some interval.

If we have it on enough machines we could make that job(the one that runs on the subset) run as part of the regular regression job as opposed to at some interval.

[mmarchini]

Ideally, we'd really like it to be run on every PR that upgrade V8 as well

I still think it's more important to run on every test run at nodejs/node-v8. Usually we open the PR to update V8 after the version branch-cut, so if the update breaks perf it might be already too late to fix it (remember: perf is not officially supported by the V8 team). Ideally the test should also run on the PR, but the priority should be to make sure it runs on nodejs/node-v8.

[joyeecheung]

Looking at nodejs/node#20783 I think it is also possible to write the test as a benchmark similar to how the http/http2 benchmarks are structured (also have external dependencies like wrk), and write a test for the benchmark controlling the parameters so that it does not take much time to run...or if we can get the benchmark job run on arbitrary base and PR, we can simply run the benchmark for v8 updates (which is also worth doing regardless of the perf test).

[mmarchini]

we can simply run the benchmark for v8 updates (which is also worth doing regardless of the perf test)

Totally agree, running benchmarks on v8 updates would be a good idea.

I think it is also possible to write the test as a benchmark

Because it's easier to setup the infra for benchmarks or there are concerns about the test speed? The test takes only a few seconds to run (should be below 10s even on slower machines).

BTW, I'm open to help install perf on the required machines.

[joyeecheung]

@mmarchini Mostly because it depends on external tools for stats, similar to how the HTTP/HTTP2 benchmarks work. Also the benchmark machines are supposed to only run one job at a time for the stability of the results, so no benchmarks should be run when running tests on them.

Although come to think of it, maybe we could put the post-mortem and perf tests in a new directory under test that don't get run by default (since the results should only change during V8 updates), and use https://ci.nodejs.org/job/node-stress-single-test/build?delay=0sec to run the subset?