coverage.nodejs.org misreporting

[chrismilleruk]

• Version: v10.0.0-pre
• Platform: tbc
• Subsystem: test, coverage

https://coverage.nodejs.org/ is reporting a significant drop in code coverage in the most recent nightly.

JS Coverage: 94.42% -> 28.80%
C++ Coverage: 92.50% -> 31.10%

Note that my local coverage results seem OK so this looks like a build / job failure.

[addaleax]

/cc @nodejs/build @mhdawson

[rvagg]

9th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/600/nodes=benchmark/console
10th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/nodes=benchmark/console

Does the output help? it has %'s that match what we're seeing on the table.

Maybe we just added a ton of code yesterday and didn't catch up with enough new tests ...

[chrismilleruk]

Those look really useful, thanks.

I had found these:
9th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/600/
10th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/
... but yours are better.

I noticed from the job status pages there that job 601 took just 16 mins instead of the usual ~31 mins.

[chrismilleruk]

I reviewed the diff of the two outputs. Aside from the obvious size difference (601 is a shorter log), there are fewer files covered and the coverage for each file is generally worse.

It seems to point towards some missing files (both from the test suite and the source files).

Is it possible to re-run the job on the CI server?

[rvagg]

maybe we just wait for today's run and see how it goes, perhaps we should delete yesterday's record if it's a simple anomaly

[rvagg]

Today's run has the same numbers as yesterdays .. something's up with the codebase I think.
I just checked on the machine and the last login was Apr 3 so I don't think this is about a software update or anything on the system.

[chrismilleruk]

Trying to repro the issue locally is proving difficult. I'm not familiar with the codebase and build setup so any pointers appreciated. I found some commands in the build repo but I can't confirm that these are the same as used by node-test-commit-linux-coverage job.
https://github.com/nodejs/build/tree/master/jenkins/scripts/coverage#coverage-job

./configure --coverage
make coverage-clean
NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python COVTESTS=test-ci make coverage -j $(getconf _NPROCESSORS_ONLN)

The above fails locally (macOS 10.13.4) with the following error:
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: unknown option character `-' in: --coverage

[chrismilleruk]

These are the commits between the last known good and the first known failure:
244af7a...d1156da

The OpenSSL changes look like a sensible suspect, there were some CI failures that have already been reviewed and discounted #19794 (comment) but it might be worth a second look?

[Trott]

@nodejs/testing

[addaleax]

Looking at the console outputs, it seems like the issue is that make test seems to fail during/after addon compilation. This is the original error message from https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/nodes=benchmark/consoleFull:

npm ERR! weird error structured-stack:1
npm ERR! weird error (function (){cov_12264jx1gp.f[26]++;cov_12264jx1gp.s[202]++;Error.prepareStackTrace=function(err,trace){cov_12264jx1gp.f[27]++;cov_12264jx1gp.s[203]++;err.stack=trace;};cov_12264jx1gp.s[204]++;Error.stackTraceLimit=Infinity;cov_12264jx1gp.s[205]++;return function structuredStack(){cov_12264jx1gp.f[28]++;cov_12264jx1gp.s[206]++;// eslint-disable-next-line no-restricted-syntax
npm ERR! weird error              ^
npm ERR! weird error 
npm ERR! weird error ReferenceError: cov_12264jx1gp is not defined
npm ERR! weird error     at structured-stack:1:14
npm ERR! weird error     at structured-stack:2:30
npm ERR! weird error     at Script.runInContext (vm.js:24:1196)
npm ERR! weird error     at Script.runInNewContext (vm.js:24:1416)
npm ERR! weird error     at runInNewContext (vm.js:28:1085)
npm ERR! weird error     at isInsideNodeModules (internal/util.js:36:44)
npm ERR! weird error     at showFlaggedDeprecation (buffer.js:26:744)
npm ERR! weird error     at new Buffer (buffer.js:41:94)
npm ERR! weird error     at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/deps/npm/node_modules/mississippi/node_modules/duplexify/index.js:6:20)
npm ERR! weird error     at Module._compile (internal/modules/cjs/loader.js:109:892)

[addaleax]

(i.e. this points to #19524 being at fault for this)