Adding known bugs to test cases

[Patrick-Poitras]

Proposal

This is a major change proposal regarding the addition of known issues into the testing suite for the purposes of detecting incidental fixes. This topic was mentioned on this discussion on Zulip that has been going around for the past few days.

In a nutshell, the proposal is to gather some known longstanding issues, and to run these tests at the end of the regular testing suite to notify users if their proposed submission fixes a known bug in the compiler. This would allow for incidental bugfixes to be noticed, which would allow for closing of issues that are incidentally fixed, and for the test cases to be added to regular tests to increase coverage. It would also ensure that temporary or draft code modifications that fix bugs incidentally are not discarded.

Roughly speaking, the changes that would need to happen are:

• The addition of known bugs to the testing suite. (A rather large undertaking, though could be done by inexperienced editors, giving them some experience in contributing to rust)
• The modification of the testing tools to allow for tests which are very likely to fail, and for a message to be displayed should this not be the case. (e.g. "Yay, you fixed an issue. Please submit it to us for consideration. (instructions)")

Mentors or Reviewers

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[jackh726]

@rustbot second

I think this makes sense for sure. There's been good support on zulip so far.

Importantly, there isn't a ton of implementation work here. Really, this MCP is mostly for "policy" - if we want these kind of tests in the test suite, we should actively encourage people to contribute MCVEs from issues. However, there might be some implementation work to allow e.g. "relaxed" rules for error annotations or stderr capture. I'm willing to mentor/review these changes.

[apiraino]

@rustbot label -final-comment-period +major-change-accepted

[nikomatsakis]

I'm a fan :) (I know I'm not supposed to comment here...)