Allow selecting a particular test or test case to run from the command line

[briancroom]

This adds the ability for test binaries using XCTMain to receive a command-line argument that specifies that only an individual test or test case should be run, as implied in this proposal:

./FooTests FooTestCase/testFoo  # Run a single test method
./FooTests FooTestCase          # Run all the tests in FooTestCase

See also swiftlang/swift-package-manager#168 which begins implementing this in swift test.

[czechboy0]

👍
What if the passed-in TestCase/Test doesn't match any existing test? I can't find anything that would warn/error out, so users might accidentally filter out all tests (test nothing), but still get a "success" return code. Or is it ensured elsewhere that at least one test runs?

[briancroom]

@czechboy0 nope, you are correct that it would run 0 tests and exit with a 0 status code. I was copying the behavior of the xctest runner app on OS X because that's what swift test would call into there, although I agree that is potentially misleading and a different behavior may be desirable.

[mike-ferris]

Very cool. The test you added for this might be expanded to run a third time with no argument to ensure that the default behavior is intact, running all the tests.

[briancroom]

Yeah I considered that, but figured that there were plenty of other funcional tests that thoroughly cover that case. Still, having the contrast present in the one test file could be nice.

[briancroom]

Added it now 😀

[modocache]

I've been thoroughly convinced by your suggestions in #43, and I think you should place these assertions as close to the source code they're testing as possible (like this: modocache@22f1d24#diff-38abae29a59be553f8b7fc357a30e6ecR14). In fact, I think we should modify all of our tests in this style (but in another PR)!

[briancroom]

For this test in particular, I don't think it's going to work to reorganize the CHECK lines, unfortunately, because of the way that this one executes the tests repeatedly in order to compare the output when running them with different test filters active.

Or is there a neat way to structure this that I haven't thought of yet?

[modocache]

See my other comment: you may be able to use a distinct check prefix for each of the three runs.

[modocache]

Wow, you make it look easy! Brilliant.

[briancroom]

Updated and squashed!

The reordered assertions worked out really well @modocache, thanks for pointing out this feature of the test infrastructure!

[briancroom]

This is ready to go as far as I am concerned. I'm not sure if there's any point in asking CI to do anything with it right now?

[modocache]

I'm under the impression that CI is mostly broken right now due to the Swift 3 migration, yeah. @mike-ferris-apple, could you ask @swift-ci to please test, just to see what happens?

Also, @mike-ferris-apple, how would you feel if I tested this using the CI presets locally, then merged? Just like the good old days before @swift-ci was enabled for this repository! 😂

[modocache]

CI issues aside, I agree that this is ready to be merged anytime!

[mike-ferris]

@swift-ci please test

[mike-ferris]

I'd be OK with merging this hand-tested since the CI seems to be in a bit of a state right now.

[modocache]

Great. I'll merge this and #68 sometime today.

[modocache]

Sorry for the slow turnaround, @briancroom. I tried running the tests on Linux but encountered the following failure, which is probably due to the Swift 3 migration:

/home/modocache/GitHub/apple/swift-corelibs-xctest/Sources/XCTest/TestFiltering.swift:63:54: error: 'split(_:maxSplit:allowEmptySlices:)' is unavailable: Please use split(separator:maxSplits:omittingEmptySubsequences:) instead
        let components = selectedTestName.characters.split("/").map(String.init)
                                                     ^~~~~
Swift.Collection:3:17: note: 'split(_:maxSplit:allowEmptySlices:)' has been explicitly marked unavailable here
    public func split(separator: Self.Iterator.Element, maxSplit: Int = default, allowEmptySlices: Bool = default) -> [Self.SubSequence]
                ^

The following change got all the tests passing for me:

diff --git a/Sources/XCTest/TestFiltering.swift b/Sources/XCTest/TestFiltering.swift
index f67d3a1..78e5191 100644
--- a/Sources/XCTest/TestFiltering.swift
+++ b/Sources/XCTest/TestFiltering.swift
@@ -60,7 +60,7 @@ private struct SelectedTest {

 private extension SelectedTest {
     init?(selectedTestName: String) {
-        let components = selectedTestName.characters.split("/").map(String.init)
+        let components = selectedTestName.characters.split(separator: "/").map(String.init)
         switch components.count {
         case 1:
             testCaseName = components[0]

On OS X, I needed to fix a build error in Foundation (swiftlang/swift-corelibs-foundation#284) in addition to applying the above change, but the tests all pass!

Ping me once you've made the above change and I'll merge--thanks!

[briancroom]

I for one welcome our new Swift 3 overlords..and am regretting not investing the time in building an updated toolchain earlier.

Regardless, I've made the change now!

[modocache]

Great! Thanks, @briancroom 💯