test: add the ability to run the tests from node in the path

[gdams]

Checklist

• make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

test

Description of change

This pr adds the ability to run tools/test.py --path. This means that instead of defaulting to out/Release we can use the node version from the path.

[jasnell]

hmmm.. LGTM I suppose. @nodejs/build

[gibfahn]

CI: https://ci.nodejs.org/job/node-test-pull-request/4889/
CI 2: https://ci.nodejs.org/job/node-test-pull-request/4919/
CI 3: https://ci.nodejs.org/job/node-test-pull-request/4963/

[gibfahn]

LGTM, this would definitely be useful for manually running tests.

[MylesBorins]

@GeorgeAdams95 what would be the purpose of this? It would seem odd to me to run tests that are different than the tests shipped with a particular binary

[gdams]

@thealphanerd this would be useful for manually re-running tests, means that rather than having to build node you could just use nvm for example

[gibfahn]

CI 4: https://ci.nodejs.org/job/node-test-commit/6993/

[gibfahn]

CI 5: https://ci.nodejs.org/job/node-test-commit/6999/
CI 6: https://ci.nodejs.org/job/node-test-commit/7003/

[gibfahn]

CI is green, any objections to this?

[jasnell]

No objections

[gibfahn]

Landed in 4198253

[evanlucas]

Added don't land on v7.x for now. I keep getting errors when running tests now:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "tools/test.py", line 141, in RunSingle
    output = case.Run()
  File "/Users/evan/dev/code/forks/WORK-node/tools/test.py", line 505, in Run
    result = self.RunCommand(self.GetCommand(), {
  File "/Users/evan/dev/code/forks/WORK-node/test/abort/../testpy/__init__.py", line 60, in GetCommand
    result = [self.config.context.GetVm(self.arch, self.mode)]
  File "tools/test.py", line 871, in GetVm
    if not ProcessOptions(options):
  File "tools/test.py", line 1429, in ProcessOptions
    options.j = int(cores) if cores is not None else multiprocessing.cpu_count()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 136, in cpu_count
    raise NotImplementedError('cannot determine number of cpus')
NotImplementedError: cannot determine number of cpus

We should probably revert until those can be addressed.

[thefourtheye]

Something like this, could fix the problem I think.

# Pastebin 2d5uLHb4
diff --git a/tools/test.py b/tools/test.py
index 18188e7b00..f553122ad5 100755
--- a/tools/test.py
+++ b/tools/test.py
@@ -852,7 +852,7 @@ class Context(object):

   def __init__(self, workspace, buildspace, verbose, vm, args, expect_fail,
                timeout, processor, suppress_dialogs,
-               store_unexpected_output, repeat):
+               store_unexpected_output, repeat, path):
     self.workspace = workspace
     self.buildspace = buildspace
     self.verbose = verbose
@@ -864,16 +864,11 @@ class Context(object):
     self.suppress_dialogs = suppress_dialogs
     self.store_unexpected_output = store_unexpected_output
     self.repeat = repeat
+    self.path = path

   def GetVm(self, arch, mode):
-    parser = BuildOptions()
-    (options, args) = parser.parse_args()
-    if not ProcessOptions(options):
-      parser.print_help()
-      return 1
-    if options.path:
-      name = find_executable("node")
-      return name
+    if self.path:
+      return find_executable("node")
     if arch == 'none':
       name = 'out/Debug/node' if mode == 'debug' else 'out/Release/node'
     else:
@@ -1577,7 +1572,8 @@ def Main():
                     processor,
                     options.suppress_dialogs,
                     options.store_unexpected_output,
-                    options.repeat)
+                    options.repeat,
+                    options.path)

[gibfahn]

Okay, so this happens when using -J with a large number of tests, and it didn't show up with CI because we don't use -J (we use -j and work out the number of cores ourselves).

Sorry for missing this in the review!