test/doctool/test-doctool-html requires internet connection?

[joyeecheung]

I got this today:

/Applications/Xcode.app/Contents/Developer/usr/bin/make -s test-doc
Running JS linter...
Running C++ linter...
Running Markdown linter on misc docs...
Running Markdown linter on docs...
=== release test-doctool-html ===
Path: doctool/test-doctool-html
Failed to add alternative version links to foo
Failed to add alternative version links to foo
Failed to add alternative version links to foo
Failed to add alternative version links to foo
/Users/joyee/projects/node/test/common/index.js:710
const crashOnUnhandledRejection = (err) => { throw err; };
                                             ^

Error: connect ETIMEDOUT 151.101.228.133:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1128:14) {
  errno: -60,
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '151.101.228.133',
  port: 443
}
Command: out/Release/node /Users/joyee/projects/node/test/doctool/test-doctool-html.js
[01:17|% 100|+   3|-   1]: Done
make[1]: *** [test-doc] Error 1
make: *** [test] Error 2

And if I turn off Wifi..

NODE_DEBUG=net out/Release/node /Users/joyee/projects/node/test/doctool/test-doctool-html.js
Failed to add alternative version links to foo
Failed to add alternative version links to foo
NET 34507: pipe false null
NET 34507: connect: find host raw.githubusercontent.com
NET 34507: connect: dns options { family: undefined, hints: 1024 }
NET 34507: _read
NET 34507: _read wait for connection
Failed to add alternative version links to foo
Failed to add alternative version links to foo
NET 34507: destroy
NET 34507: close
NET 34507: close handle
/Users/joyee/projects/node/test/common/index.js:710
const crashOnUnhandledRejection = (err) => { throw err; };
                                             ^

Error: getaddrinfo ENOTFOUND raw.githubusercontent.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:60:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'raw.githubusercontent.com'
}

Has there been any changes that add this new requirement? It seems weird to require internet connection to run the doc tests.

[richardlau]

Has there been any changes that add this new requirement? It seems weird to require internet connection to run the doc tests.

#27661

[joyeecheung]

cc @nodejs/testing

@richardlau Thanks for the pointer. Can we use something in the file system instead? (That is to go back to what #27661 originally did, cc @BridgeAR ). In my opinion, the default test suites should not require an Internet connection , otherwise it should be placed under test/internet - not every access point guarantees stable, unpolluted connection to raw.githubusercontent.com and requiring the connection to it in the default test suite leads to false failures that have nothing to do with the code.

[richardlau]

It's not the test suite but the actual doctool that is making the connection (the test added by that PR was placed in internet but obviously with hindsight existing tests that ran the tool were affected). Also doctool tests are not in the default set of test suites.

I guess the question is whether we care whether or not that the API docs can point to a newer release line? e.g. the 12.x docs pointing to the 13.x equivalent in the "View another version" drop down -- the issue with the file system is the changelog in the 12.x branch will not have a reference to 13.x (even after it is released as we don't backport those kinds of changes).

Open to any suggestions. The original aim of #27661 was to reduce the manual effort in keeping the version picker in the docs accurate.

[Trott]

Perhaps unrelated, perhaps related, I'm not sure: The same test is now perma-red on win2008r2 in CI all of a sudden. Not sure why or what's up. At first I thought it might be internet based on this issue, but the problem looks different in that there's no ETIMEDOUT or ENOTFOUND.

Example: https://ci.nodejs.org/job/node-test-binary-windows-2/3517/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=addons/testReport/(root)/test/doctool_test_doctool_html/

@nodejs/build @nodejs/platform-windows

[Trott]

Perhaps unrelated, perhaps related, I'm not sure: The same test is now perma-red on win2008r2 in CI all of a sudden. Not sure why or what's up. At first I thought it might be internet based on this issue, but the problem looks different in that there's no ETIMEDOUT or ENOTFOUND.

Example: https://ci.nodejs.org/job/node-test-binary-windows-2/3517/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=addons/testReport/(root)/test/doctool_test_doctool_html/

@nodejs/build @nodejs/platform-windows

Perhaps this is a different issue possibly caused by f9caee9 somehow. (Only suggesting it because it's the only commit that's landed in a long time that's touched the doc tool at all, but it does seem unlikely now that I look more closely. But who knows?)

[Trott]

Nope, reverting f9caee9 doesn't fix anything... :-(

[Trott]

Perhaps unrelated, perhaps related, I'm not sure: The same test is now perma-red on win2008r2 in CI all of a sudden. Not sure why or what's up. At first I thought it might be internet based on this issue, but the problem looks different in that there's no ETIMEDOUT or ENOTFOUND.

Example: https://ci.nodejs.org/job/node-test-binary-windows-2/3517/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=addons/testReport/(root)/test/doctool_test_doctool_html/

@nodejs/build @nodejs/platform-windows

I think it's flaky for unrelated reasons. Will open something else if/when I get to the bottom of it.

[joyeecheung]

I am preparing for the code and learn tomorrow and I noticed that this may cause issues with the conference WiFi where connections to raw.githubusercontent.com result in ETIMEOUT. I'll ask the participants to use make test-only for now (even then this still shows up but it does not stop the build)

[richardlau]

PR: #30214