testament: `failureAllowed:true` to allow issue reporters to add test cases, and get notified upon "accidental" fix

[timotheecour]

An issue reporter (or anyone else) can add tests/issues/t12345.nim to the issue #12345 he's reporting (via a PR); it'd look like this:

discard """
# any other spec field allowed, eg `cmd`, `errormsg`, `nimout` etc
failureAllowed: true
"""

# write this the way you would write any test case, eg:
doAssert thisShouldWorkButDoesnotYet() 

The semantic is different from disabled:true, as the test shall be run by testament, but won't cause a CI failure if it fails.
If the test succeeds, however, we shall issue a notification (for now, can just be appropriate logging that's easily identifiable but there are other ways, eg email, using travis API, etc) so we can act on it by:

• removing failureAllowed: true
• closing the corresponding github issue and adding a comment in the PR that fixed that issue "by accident"

benefits

• issue reporters who care about getting their issue fixed faster can send a PR to add a test case
• it removes some of the load from the person writing the fix for the issue (most important point)
• it allows getting notified when some old issue was fixed "by accident", and helps reducing issue count
• it allows identifying what PR caused the fix "by accident"

example

for a recent example of accidental fix: see nim-lang/Nim#10169

in reply to nim-lang/Nim#7976 (comment) from @@kaushalmodi , I verified that 083129286349ba440018cff1ed20172b675b84fe is the commit that fixed #7976 (see my corresponding PR #10071)

note

Other people have suggested something like this on the forum(IRC) but I don't remember where.

[timotheecour]

related: see flakyAssert which I added in testament/lib/stdtest/testutils.nim

so instead of a new failureAllowed: true spec (which is not flexible, not fine grained etc), we can now either use
flakyAssert(thisShouldWorkButDoesnotYet(), notifySuccess = true) anywhere in some existing test file, or, add a dedicated API for this use case, eg:

unfixedAssert(thisShouldWorkButDoesnotYet()) # issue #123

in either case, what's missing is the notification part when a test gets "accidentally fixed" in some PR, which is not hard to do (eg email, IRC, warning shown in CI summary etc)
then converting to a regular assert is easy, simply change unfixedAssert to doAssert, that's it

I'm leaving this issue open until notification part gets done

[github-actions]

This RFC is stale because it has been open for 1095 days with no activity. Contribute a fix or comment on the issue, or it will be closed in 7 days.