parallel_testsuite.py: Use multiprocessing pool

[sbc100]

Use the Pool seems to address the longstanding issue we have had with interrupting the test suite using crtl-C.

Its also a lot less code.

Also, limit the pools size to the number of tests being run. No need to start 56 processes if we are just running 2 tests.

I measured the performance of the running the core2 test suite both before and after this change and it was not effected.

See: #18213
Supersedes: #11299

[juj]

Does this part get tested on Windows on CircleCI? Python has very exotic rules on Windows for what it can pickle and what it cannot, since Windows does not support fork().

[RReverser]

Just tried this locally on Windows and it seems to cause some new issues instead. Seeing a lot of

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".\test\runner.py", line 432, in <module>
    sys.exit(main(sys.argv))
  File ".\test\runner.py", line 421, in main
    num_failures = run_tests(options, suites)
  File ".\test\runner.py", line 300, in run_tests
    res = testRunner.run(suite)
  File "C:\Python38\lib\unittest\runner.py", line 176, in run
    test(result)
  File "C:\Python38\lib\unittest\suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "C:\Users\me\Documents\emscripten\test\parallel_testsuite.py", line 53, in run
    results = [r.get() for r in results]
  File "C:\Users\me\Documents\emscripten\test\parallel_testsuite.py", line 53, in <listcomp>
    results = [r.get() for r in results]
  File "C:\Python38\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
PermissionError: [Errno 13] The process cannot access the file because it is being used by another process: 'C:\\Users\\me\\AppData\\Local\\Temp\\tmpgkue8p3_'

Looks like parallel processes are now somehow stepping on each others toes?

[RReverser]

Ah right, I now see that ci/circleci: test-windows shows the same issue.

Does this part get tested on Windows on CircleCI?

@juj I suppose that answers your question :)

[sbc100]

I've very excited about this change.. the inability to interrupt the parallel running without spamming stderr has annoyed me for so many years now.

[sbc100]

Windows now passes!

[RReverser]

I don't see any difference on Windows, and actually the VSCode issue I described in #18213 seems to apply in regular terminal as well, it's just sporadic and doesn't always reproduce :(

Given all those Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? in the log, I suspect the problem is that emcc.bat files are executed via cmd.exe in, what it thinks is, interactive mode, so each of those cmd.exe instances waits for confirmation from the user.

There should be some way to tell it it's not interactive (pass empty stdin at Python level if it's not doing that yet?)...

[kripken]

Nice simplification - seems worthwhile even if doesn't fully resolve the issue.

Probably worth manual testing to make sure this does what is expected, both in terms of the issue and also in not silently missing test failures. On Linux both look good to me locally.

[RReverser]

pass empty stdin at Python level if it's not doing that yet?

For a quick check, I tried just doing out-null | python .\test\runner.py ... (equivalent of </dev/null) and, well, it at least cancels, even though the output is still bit verbose:

Process SpawnPoolWorker-2:test_pthread_proxying_modularize (test_core.core0) ... ERROR
Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)?
    self._writer.send_bytes(obj)rk (test_core.core0) ... ERROR
  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes
Process SpawnPoolWorker-3:  File "C:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap
Terminate batch job (Y/N)?     self.run()onnection.py", line 280, in _send_bytes
  File "C:\Python38\lib\multiprocessing\process.py", line 108, in run
  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker
test_pthread_specific (test_core.core0) ... ERROR    put((job, i, (False, wrapped)))test_pthread_setspecific_mainthread (test_core.core0) ... ERRORProcess SpawnPoolWorker-6:
    self._writer.send_bytes(obj)  File "C:\Python38\lib\multiprocessing\pool.py", line 131, in worker
Traceback (most recent call last):st_core.core0) ... ERRORtest_pthread_threa
  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytess.py", line 365, in put
    put((job, i, result))Process SpawnPoolWorker-7:
    self._send_bytes(m[offset:offset + size])py", line 131, in worker
  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put
    put((job, i, result))Process SpawnPoolWorker-8:  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes    self._writer.send_bytes(obj)
  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)  File "C:\Python38\lib\multiprocessing\connection.py", Traceback (most recent call last):BrokenPipeError: [WinError 232] The pipe is being closed
  File "C:\Python38\lib\multiprocessing\pool.py", line 131, in worker  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes
Traceback (most recent call last):et + size])
Process SpawnPoolWorker-4:

  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes    put((job, i, result))

  File "C:\Python38\lib\multiprocessing\pool.py", line 131, in workerProcess SpawnPoolWorker-5:
  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytesthon38\lib\multiprocessing\queues.py", line 365, in put


    put((job, i, result))Traceback (most recent call last):    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
    self._writer.send_bytes(obj)BrokenPipeError: [WinError 232] The pipe is being closed  File "C:\Python38\lib\multiprocessing\pool.py", line 131, in workerBrokenPipeError: [WinError 232] Th
  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytesDuring handling of the above exception, another exception occurred:Terminate batch job (Y/N)?     self._writer.send_bytes(obj)    put((job, i, result))    put((job, i, result))During handling of the above exception, another exception occurred:
    self._send_bytes(m[offset:offset + size])  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes
    self._writer.send_bytes(obj)    self._writer.send_bytes(obj)rocessing\queues.py", line 365, in put  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in putTraceback (most recen
  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes
tstrap    self.run()i.WriteFile(self._handle, buf, overlapped=True)bootstrap    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [WinError 232] The pipe is being closedline 200, in send_bytes  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes
essing\connection.py", line 280, in _send_bytes  File "C:\Python38\lib\multiprocessing\process.py", line 108, in run    self._send_bytes(m[offset:offset + size])
During handling of the above exception, another exception occurred:    self._target(*self._args, **self._kwargs)  File "C:\Python38\lib\multiprocessing\process.py", line 108, in run
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)n _send_bytes
BrokenPipeError: [WinError 232] The pipe is being closedline 280, in _send_bytes
Traceback (most recent call last):re0) ... ERROR
    self._target(*self._args, **self._kwargs)py", line 136, in worker
  File "C:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker
During handling of the above exception, another exception occurred:    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)BrokenPipeError: [WinError 232] The pipe is being closed
   put((job, i, (False, wrapped)))
BrokenPipeError: [WinError 232] The pipe is being closed
    self.run()

Traceback (most recent call last):During handling of the above exception, another exception occurred:
  File "C:\Python38\lib\multiprocessing\process.py", line 108, in run  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put
    self._writer.send_bytes(obj)):ssing\queues.py", line 365, in put



During handling of the above exception, another exception occurred:
  File "C:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap    self._writer.send_bytes(obj)    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):ssing\connection.py", line 200, in send_bytes
  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker    self.run()  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes    self.run()
  File "C:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap    put((job, i, (False, wrapped)))
Process SpawnPoolWorker-1:  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes, in run
  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in workert(*self._args, **self._kwargs)    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):    put((job, i, (False, wrapped)))    put((job, i, (False, wrapped)))    self._target(*self._args, **self._kwargs)    self._writer.send_bytes(obj)BrokenPipeError: [WinError 232] The pipe is being closedbuf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed  File "C:\Python38\lib\multiprocessing\pool.py", line 131, in worker  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put
  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes    self._writer.send_bytes(obj)
    self._send_bytes(m[offset:offset + size])    self._writer.send_bytes(obj)    put((job, i, (False, wrapped)))  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put
  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put    self._send_bytes(m[offset:offset + size])  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes    self._writer.send_bytes(obj)
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)    self._writer.send_bytes(obj)  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [WinError 232] The pipe is being closed  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes    self._send_bytes(m[offset:offset + size])    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)BrokenPipeError: [WinError 232] The pipe is being closed    self._send_bytes(m[offset:offset + size])  File "C:\Python38\libBrokenPipeError: [WinError 232] The pipe is being closeds
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)n _send_bytes

    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)BrokenPipeError: [WinError 232] The pipe is being closed
BrokenPipeError: [WinError 232] The pipe is being closed
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Python38\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python38\lib\multiprocessing\pool.py", line 136, in worker
    put((job, i, (False, wrapped)))
  File "C:\Python38\lib\multiprocessing\queues.py", line 365, in put
    self._writer.send_bytes(obj)
  File "C:\Python38\lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "C:\Python38\lib\multiprocessing\connection.py", line 280, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed

[sbc100]

yes, this most certainly helps address the issue on linux. I still sometimes see traceback, it its not O(N) in the number of subprocesses which is what was killing me before.

[juj]

Given all those Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? in the log, I suspect the problem is that emcc.bat

That is an unfortunate behavior of Windows batch files, I don't think there is a way to get around that :/

But awesome in general to see that the runner shouldn't hang anymore when aborting tests!

[RReverser]

That is an unfortunate behavior of Windows batch files, I don't think there is a way to get around that :/

As mentioned above, it is possible to bypass it if it knows it's running in non-interactive mode - aka stdin is not propagated from parent. It will still print "Terminate batch job (Y/N)" but it won't actually wait for user to type Y/N and instead exit immediately, which is what we want here.

[sbc100]

Sadly it might be hard to know know a-prior if the compiler needs to read from stdin.

[RReverser]

Sadly it might be hard to know know a-prior if the compiler needs to read from stdin.

But, in the tests, we always know the test subprocesses don't need to read from stdin? Is there no way to configure the Pool to redirect stdin from empty stream for subprocesses?

[sbc100]

Sadly it might be hard to know know a-prior if the compiler needs to read from stdin.

But, in the tests, we always know the test subprocesses don't need to read from stdin? Is there no way to configure the Pool to redirect stdin from empty stream for subprocesses?

True, the python pool of processes don't need to read from stdin. But those are not .bat processes, they (I think) are python.exe processes.

I think the Terminate batch job (Y/N) messages come from emcc.bat which happens to be a very common subprocess of most test processes. We do have some tests (admittedly very few) that do pass stuff to emcc via stdin. e.g. emcc -c - (which means compile from stdin).

e.g.:

emscripten/test/test_other.py

Lines 10710 to 10717 in ce4c405

	def test_stdin_compile_only(self):
	# Should fail without -x lang specifier
	src = read_file(test_file('hello_world.cpp'))
	err = self.expect_fail([EMCC, '-c', '-'], input=src)
	self.assertContained('error: -E or -x required when input is from standard input', err)
	self.run_process([EMCC, '-c', '-o', 'out.o', '-x', 'c++', '-'], input=src)
	self.assertExists('out.o')

[RReverser]

I think the Terminate batch job (Y/N) messages come from emcc.bat

Indeed.

We do have some tests (admittedly very few) that do pass stuff to emcc via stdin. e.g. emcc -c - (which means compile from stdin).

Right, but those tests already pipe stdin from another input, so if we set a global stdin for Python subprocesses to empty stream, they won't care either way.

That is, what I'm suggesting is: Python Pool should set stdin to empty stream. Then, all subprocesses of Pool, whether tests or emcc.bat, will recursively inherit this empty stdin by default, but they can still override it to a custom one so these tests that have input=src should still work as expected.

[sbc100]

I think the Terminate batch job (Y/N) messages come from emcc.bat

Indeed.

We do have some tests (admittedly very few) that do pass stuff to emcc via stdin. e.g. emcc -c - (which means compile from stdin).

Right, but those tests already pipe stdin from another input, so if we set a global stdin for Python subprocesses to empty stream, they won't care either way.

That is, what I'm suggesting is: Python Pool should set stdin to empty stream. Then, all subprocesses of Pool, whether tests or emcc.bat, will recursively inherit this empty stdin by default, but they can still override it to a custom one so these tests that have input=src should still work as expected.

If that works, and fixes your issue, that sounds like a great solution!

[RReverser]

If that works, and fixes your issue, that sounds like a great solution!

Indeed... I just don't know that much about Python multiprocessing to know how to configure stdin in said way without digging in 😅