Add a place for reproducing current bugs

[cjihrig]

Currently, Node's issue tracker is closing in on 500 open issues. A lot of issues seem to get forgotten about. Then, when someone gets around to triaging, he/she has to try to reproduce the issue to see if it is still valid. Sometimes there is a reproducible test case. Sometimes there isn't. Sometimes a bug is still valid. Sometimes the bug has been fixed and the issue just slipped through the cracks.

I'm proposing adding a directory of tests that reproduce as many issues as possible (preferably one script per bug). I think this would be nice because, as bugs are fixed, the scripts could be moved out of this purgatory directory and into the tests that are run by the CI. It would also make it simpler to triage open issues - one command to see what issues still exist.

Thoughts?

[geek]

This is a good direction... will make detecting fixed issues easier. You will be able to periodically run the /broken-test folder to find when an issue is already corrected.

It also helps people reporting issues to feel empowered about submitting a PR with a broken test to demonstrate the issue.

The solution should also lighten the load of issue sizes as well, since reproducing code can go into a real js file.

[cjihrig]

ping...

[santigimeno]

I really like the idea. What would be the workflow? Should the reporters be encouraged to submit the PRs?

[cjihrig]

My thoughts are that reporters would be encouraged, but not required to, submit the failing test cases. Since it would have to go through the normal PR process, outside reporters might not want to be bothered. It would be more of a helper for collaborators in triaging issues and writing tests. From our point of view, the workflow would just be to PR failing tests for existing issues (assuming the CTC is OK with it).

[santigimeno]

Ok, it makes sense to me. Thanks

From the CI point of view, just running make jslint on the new folder right?

[cjihrig]

Yes, linting at the most, since the tests would be expected to fail. Unless there is some way to run the tests and verify that they fail. That is beyond what I originally had in mind though.

[orangemocha]

It sounds like an interesting idea. I wonder if making it go through the regular PR/review process will discourage people from submitting the test case.

We could lower the bar a bit and consider these more repro-cases than broken-tests. By that I mean that a repro-case doesn't have to be perfect according to the test guidelines in order to be useful. Of course, we can have both.

For well written tests that are known to fail, we could also include them in the regular test suite and leverage the status files. If you tag it as FAIL, the test is expected to fail and you get an error in the tap report if it actually passes.

[cjihrig]

I wonder if making it go through the regular PR/review process will discourage people from submitting the test case.

I'd rather discourage people from submitting a PR. They can just submit an issue, and a collaborator can make a PR. I think we are a lot less likely to get buy in from people if we try to lower the bar on PRs into nodejs/node.

[orangemocha]

Makes sense. Less well-formed repro cases - if we wanted such a thing - could go into a separate repo.

[cjihrig]

@nodejs/ctc would you be OK with this proposal?

[Trott]

Without getting too much into process/implementation details, I like this idea a lot.

I also think that creating tests (without fixes) from bug reports would make an excellent source of good-first-contribution possibilities.

Feel free to add the wg-agenda tag if you want to talk about this specifically at the Testing WG meeting tomorrow.

[ChALkeR]

Looks like a great idea to me.

Moreover, perhaps it should be turned into a desireable way of dealing with bug reports — after a testcase is built (and without a valid testcase bugs are usually closed after some amount of time), it could be pushed into the broken-tests directory, and if no testcase could be built, one should try better to explain why this is actually a bug and why a testcase is impossible.

To achive the goal here, perhaps it would be possible to lower the bar a bit, because else it could delay merging in new broken-tests, and we don't want those to keep hanging as PRs, correct?

Those tests could be reused and moved to actual tests on a case-by-case basis in the bugfix PRs, perhaps with some cleanup. Note that often finding the source of the bug and/or fixing it gives clearer understanding what exactly should be tested, so the tests will get rewritten from scratch in some cases.

This will create some extra pr/commit noise, but it doesn't look like a problem to me.

[cjihrig]

This exists now (nodejs/node#5528). Just need to populate the folder now. Closing this issue.