doc: How to run just the .out tests?

[GeoffreyBooth]

I’ve done some digging, and I’m sure if I kept looking I’d eventually find it, but could someone please tell me how to run just the tests that check expected output against the .out files? Those tests are annoying to update, as it’s a lot of trial-and-error to get the new output in, and it’s tedious to have to run the full make test for every attempt.

This feels like something that should be documented in https://github.com/nodejs/node/blob/master/BUILDING.md#running-tests.

[Trott]

These would be tests in test/messages and test/pseudo-tty, right? If so, you can run them with tools/test.py messages pseudo-tty. If you know which one specifically you want to run, you can specify it: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs.

Does that help?

[GeoffreyBooth]

Yes! Thank you, that’s what I was looking for.

I was frustrated by the fact that when these tests fail, the output tells me the command the test runner ran to generate the output, not to compare the output. In other words, if I edit test/message/test-esm-loader-obsolete-hooks.out so that it causes the test to fail, the output of this failing test includes this:

Command: out/Release/node --no-warnings --throw-deprecation --experimental-loader ./test/fixtures/es-module-loaders/hooks-obsolete.mjs /Users/Geoffrey/Sites/node/test/message/test-esm-loader-obsolete-hooks.mjs

Which for other types of tests is a one-liner to re-run that particular test; but for these “compare to .out file” tests, it’s not.

Anyway because these tests are somewhat unusual, I think a callout explaining them in the docs might help others; it would at least have helped me. It’s something I’ve been wondering about for years.

[Trott]

Anyway because these tests are somewhat unusual, I think a callout explaining them in the docs might help others; it would at least have helped me. It’s something I’ve been wondering about for years.

Since this has been a pain point for you and that doc is where you looked for information and didn't find it, I am +1 on adding the information there. Do you want to open a PR for that or would you prefer I do it?

[GeoffreyBooth]

Do you want to open a PR for that or would you prefer I do it?

I don't understand these tests well, so I think it would be better if you wrote this.

Could we add something to the output where if the command that was run is from one of these tests, another line follows the Command like with something like As part of: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs?

[Trott]

I don't understand these tests well, so I think it would be better if you wrote this.

Included in #40933.

Could we add something to the output where if the command that was run is from one of these tests, another line follows the Command like with something like As part of: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs?

I think the way to do this would be:

Move the compare-output-to-.out-file stuff out of test/message/testcfg.py and test/pseudo-ttyp/testcfg.py, and into a module in test/common that all these tests load. Perhaps the module would add a listener for the 'exit' event on process and check the output there. (It would probably load the expected output at an earlier time, perhaps when the module is first required/imported.)

Then running the test directly with node rather than test.py would still do the comparison, so the existing displayed command would work the way someone might expect.

@nodejs/testing What do you think?

[GeoffreyBooth]

It seems to me that maybe all these .out-file tests for error messages, at least for the ESM subsystem that I work on, could have been written like https://github.com/nodejs/node/blob/master/test/es-module/test-esm-invalid-extension.js where the test itself is responsible for spawning the child process and validating the error message. As in, perhaps it was a mistake to use the .out file pattern for these ESM tests, and we should refactor them to be like test-esm-invalid-extension.js. If we did that, running python tools/test.py -J --mode=release es-module really would run all the ESM tests. Any thoughts on this @Trott or others? cc @JakobJingleheimer

[Trott]

It seems to me that maybe all these .out-file tests for error messages, at least for the ESM subsystem that I work on, could have been written like https://github.com/nodejs/node/blob/master/test/es-module/test-esm-invalid-extension.js where the test itself is responsible for spawning the child process and validating the error message. As in, perhaps it was a mistake to use the .out file pattern for these ESM tests, and we should refactor them to be like test-esm-invalid-extension.js. If we did that, running python tools/test.py -J --mode=release es-module really would run all the ESM tests. Any thoughts on this @Trott or others? cc @JakobJingleheimer

I have no opinion because I'm not familiar with the friction involved. No objection from me, certainly! If it simplifies things for you and others, then that's reason enough for me.

[JakobJingleheimer]

When I first started, these .out file tests were extremely confusing. As an implementor, these are very tedious as they break whenever the wind blows despite no substantive changes.

I think moving to an approach like in the test Geoffrey cited would be a huge improvement 🙏

[GeoffreyBooth]

Fixed by #40933.