Repository navigation
The assertion module should signals failures through messages to the test runner #52033
Description
Activity
- changed the title
[-]The test runner and the assertion modules should be more cohesive[/-][+]The assertion module should signals failures through messages to the test runner[/+]on Mar 10, 2024 There has been no activity on this feature request for 5 months. To help maintain relevant open issues, please add the never-stale
Issues and PRs exempt from automated stale handling. label or close this issue if it should be closed. If not, the issue will be automatically closed 6 months after the last non-automated comment.
For more information on how the project manages feature requests, please consult the feature request management document.still applies
- added a commit that references this issue
on Jan 4, 2025 - added a commit that references this issue
on Jan 13, 2025 2 remaining items
- added a commit that references this issue
on Feb 4, 2025 - added a commit that references this issue
on Mar 13, 2025 It doesn't resolve this issue completely. See #56434 (comment).
It doesn't resolve this issue completely. See #56434 (comment).
It doesn't resolve it at all, unless there's a way to signal a failure to the text runner/context without resorting to an exception. And your comment about a
testContext.prototype.fail()missing points at that direction.Shouldn't be we reopen it?
For instance, if the custom assertions can signal success or failure by returning true or false...
since the "exception channel" is also used by the program under test which may catch the exceptions generated by failing assertions
Shouldn't test assertions about the program under test occur outside the program under test? Using your terminology, exceptions thrown within the program belong to the program's "exception channel", while those outside the program but within the wrapping test function belong to the test's exception channel, with any uncaught exceptions in the program's exception channel flowing out to the test's exception channel.
For example, compare the four tests in this modified version of your first example:
import {test} from "node:test"; import assert from "node:assert"; function function_under_test(callback, value) { let r = 0; try { r = callback(value); } catch (error) { //log error... } if (r === undefined) throw 'POOF!' //continue executing operations... } test("test should PASS", function () { function_under_test(function (v) { assert.fail("BOOM!"); }, 4); }); test("test should FAIL", function () { function_under_test(function (v) { assert.fail("BOOM!"); }, 4); assert.fail("BOOMERANG!"); }); test("test should FAIL", function () { function_under_test(function (v) { return undefined }, 4); }); test("test should PASS", function () { assert.throws( function_under_test(function (v) { return undefined }, 4), '/POOF!/' ); });
Likewise, in your second example, why would
function_under_test, presumably a black box meant to work outside of a testing context, referencetestContext?test("weird test", function (testContext) { function_under_test(function (v) { testContext.assert.fail("BOOM!"); }, 4); });
@cjihrig wouldn't you agree?
Shouldn't test assertions about the program under test occur outside the program under test? Using your terminology, exceptions thrown within the program belong to the program's "exception channel", while those outside the program but within the wrapping test function belong to the test's exception channel, with any uncaught exceptions in the program's exception channel flowing out to the test's exception channel.
Not really. There's only one exception channel, the issue here is that both the program under test and the test module are using it, with possible (like in the sample I provided) collisions.
Don't be fooled by the example which is using a simple assert.fail, a more realistic assertion may involve the v variable, and the function under test may be an async function). It's completely plausible to inject assertions in a callback function. There's no "outside" or "inside".You're right in that even if
function_under_testis a blackbox that takes a callback, the test could pass it a callback meant for testing. That said...There's no "outside" or "inside".
Still not sure I agree. Here
function_under_testis async and the assert that needs to fail the test works because it was moved outside:async function function_under_test(callback, value) { let r = 0; try { r = callback(value); } catch (error) { //log error... } //continue executing operations... } test("test will FAIL", function (testContext) { let failed = false await function_under_test(function (v) { failed = true }, 4); if (failed) testContext.assert.fail("BOOM!"); });
Still not sure I agree. Here
function_under_testis async and the assert that needs to fail the test works because it was moved outside:Looks like a hack to me 😺
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.still relevant
What is the problem this feature will solve?
The assertion module signals failures via exceptions. This may causes some subtle issues during program testing, since the "exception channel" is also used by the program under test which may catch the exceptions generated by failing assertions resulting in weird test behavior.
test passes...
What is the feature you are proposing to solve the problem?
When in a test runner context provide the assertion module via the testContext object:
and signal the failures via messages between testContext and assert methods.
What alternatives have you considered?
No response