[RFC] Who tests the tests? #3
Conversation
|
Testing the test framework is absolutely something we need to get started. Let's discuss this on the |
|
Gotcha! I sent an email to the mailing list. Feel free to close this, or we can keep it open as I push more commits to it. Thanks! 💯 |
|
Thanks for your interest, and I agree with Tony that getting testing of the testing framework should be a strong goal. For the pull requests, I think we would rather see small targeted and orthogonal changes. For example, I'd be happy to take the change to make the indent prefs explicit in the Xcode project immediately. The notion of reporters can be treated separately, and if there's other things you want to propose and discuss we can treat those separately as well. |
modocache
force-pushed
the
who-tests-the-tests
branch
from
8303f5d to
11f1fbc
Compare
Thanks for clarifying, @mike-ferris-apple! I was worried you might consider the Xcode project file change too trivial to warrant a pull request, but now that I see that's not the case, I've submitted it as #5. |
Not everyone wants to be notified of test results in the same way. Some people prefer a concise view that fits on a single line, with dots indicating success and "F" to signify failure: ".....FF..F" Some prefer a detailed view, containing data on how long each test took to run, etc. Objects that conform to the reporter protocol are responsible for formatting output related to test execution. Besides providing a way to customize test output, reporters are essential for another reason: as a way to unit test XCTest itself. Since XCTest currently prints to stdout, it's difficult to test it without running it and inspecting its output. A forthcoming commit will add tests that use a test double as a reporter, making assertions on what it output by XCTest.
modocache
force-pushed
the
who-tests-the-tests
branch
from
11f1fbc to
76107c8
Compare
|
I'm going to go ahead and close this pull request for now while the discussion continues on the mailing list. |
|
Good call, thanks! |
First of all, thanks so much for open-sourcing this!! 💘
I noticed this project doesn't have GitHub issues enabled, so I hope you'll consider this an RFC where we can discuss the issue further. Please let me know if I should email the swift-corelibs-dev mailing list instead.
I noticed that this project isn't itself unit tested. It'd be nice to add some tests, since those would enable us to work on it while avoiding regressions.
I could think of several ways to do test this project. The simplest would be running this version of XCTest on some sample programs and examining the output. Those programs would call
XCTMain()from within theirmain()function. Test output would be printed to stdout via theprint()function. Tests for this project would then verify that what we expected was indeed printed to stdout.Another approach would be what's currently in this pull request: allowing consumers of XCTest to pass a list of reporters to their
XCTMain()function. These reporters would respond to a set of messages, similar toXCTestObservation.hin the Apple version of XCTest. From within this project's unit tests, we could then pass aTestReporterthat captures the messages it's sent. We can then verify it was sent the expected messages.Let me know your preferred approach, or any other thoughts you may have. Thanks!