XCTestObservation and XCTestObservationCenter #69
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| // This source file is part of the Swift.org open source project | ||
| // | ||
| // Copyright (c) 2016 Apple Inc. and the Swift project authors | ||
| // Licensed under Apache License v2.0 with Runtime Library Exception | ||
| // | ||
| // See http://swift.org/LICENSE.txt for license information | ||
| // See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors | ||
| // | ||
| // | ||
| // ObjectWrapper.swift | ||
| // Utility type for adapting implementors of a `class` protocol to Hashable | ||
| // | ||
|
|
||
| /// A `Hashable` representation of an object and its ObjectIdentifier. This is | ||
| /// useful because Swift classes aren't implicitly hashable based on identity. | ||
| internal struct ObjectWrapper<T>: Hashable { | ||
| let object: T | ||
| let objectIdentifier: ObjectIdentifier | ||
|
|
||
| var hashValue: Int { return objectIdentifier.hashValue } | ||
| } | ||
|
|
||
| internal func ==<T>(lhs: ObjectWrapper<T>, rhs: ObjectWrapper<T>) -> Bool { | ||
| return lhs.objectIdentifier == rhs.objectIdentifier | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,7 +26,14 @@ public typealias XCTestCaseEntry = (testCaseClass: XCTestCase.Type, allTests: [( | |
|
|
||
| public class XCTestCase { | ||
|
|
||
| /// The name of the test case, consisting of its class name and the method name it will run. | ||
| /// - Note: FIXME: This property should be readonly, but currently has to be publicly settable due to a | ||
| /// toolchain bug on Linux. To ensure compatibility of tests between | ||
| /// swift-corelibs-xctest and Apple XCTest, this property should not be modified. | ||
|
There was a problem hiding this comment. Oh wow, interesting... It would be great to reference a JIRA issue here. ☝️ |
||
| public var name: String | ||
|
|
||
| public required init() { | ||
| name = "\(self.dynamicType).<unknown>" | ||
| } | ||
|
|
||
| public func setUp() { | ||
|
|
@@ -72,25 +79,34 @@ extension XCTestCase { | |
| } | ||
|
|
||
| internal static func invokeTests(tests: [(String, XCTestCase throws -> Void)]) { | ||
| let observationCenter = XCTestObservationCenter.sharedTestObservationCenter() | ||
|
|
||
| var totalDuration = 0.0 | ||
| var totalFailures = 0 | ||
| var unexpectedFailures = 0 | ||
| let overallDuration = measureTimeExecutingBlock { | ||
| for (name, test) in tests { | ||
| let testCase = self.init() | ||
| testCase.name = "\(testCase.dynamicType).\(name)" | ||
|
|
||
| var failures = [XCTFailure]() | ||
| XCTFailureHandler = { failure in | ||
| observationCenter.testCase(testCase, | ||
| didFailWithDescription: failure.failureMessage, | ||
| inFile: String(failure.file), | ||
| atLine: failure.line) | ||
|
There was a problem hiding this comment. The description we pass to the observers is different from the description that's printed by Could we have Of course, this could be something we take care of later. Let me know if you'd rather we address this in later commits. There was a problem hiding this comment. I've cleaned this up now 👍 |
||
|
|
||
| if !testCase.continueAfterFailure { | ||
| failure.emit(testCase.name) | ||
| fatalError("Terminating execution due to test failure", file: failure.file, line: failure.line) | ||
| } else { | ||
| failures.append(failure) | ||
| } | ||
| } | ||
|
|
||
| XCTPrint("Test Case '\(testCase.name)' started.") | ||
|
|
||
| observationCenter.testCaseWillStart(testCase) | ||
|
There was a problem hiding this comment. Here's a crazy idea: could we define an I'm not sure off the top of my head whether Apple XCTest does this, but I think it'd be wild! 😎 There was a problem hiding this comment. As it turns out, Apple XCTest does exactly that (well, not exactly, because the logging uses the legacy I would like to make that refactor, yes. In fact, one reason why I wrote the observer in the functional test to keep track of state in properties instead of just |
||
|
|
||
| testCase.setUp() | ||
|
|
||
|
|
@@ -107,20 +123,22 @@ extension XCTestCase { | |
| testCase.failIfExpectationsNotWaitedFor(XCTAllExpectations) | ||
| XCTAllExpectations = [] | ||
|
|
||
| observationCenter.testCaseDidFinish(testCase) | ||
|
|
||
| totalDuration += duration | ||
|
|
||
| var result = "passed" | ||
| for failure in failures { | ||
| failure.emit(testCase.name) | ||
| totalFailures += 1 | ||
| if !failure.expected { | ||
| unexpectedFailures += 1 | ||
| } | ||
| result = failures.count > 0 ? "failed" : "passed" | ||
| } | ||
|
|
||
| XCTPrint("Test Case '\(testCase.name)' \(result) (\(printableStringForTimeInterval(duration)) seconds).") | ||
| XCTAllRuns.append(XCTRun(duration: duration, method: testCase.name, passed: failures.count == 0, failures: failures)) | ||
| XCTFailureHandler = nil | ||
| } | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| // This source file is part of the Swift.org open source project | ||
| // | ||
| // Copyright (c) 2016 Apple Inc. and the Swift project authors | ||
| // Licensed under Apache License v2.0 with Runtime Library Exception | ||
| // | ||
| // See http://swift.org/LICENSE.txt for license information | ||
| // See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors | ||
| // | ||
| // | ||
| // XCTestObservation.swift | ||
| // Hooks for being notified about progress during a test run. | ||
| // | ||
|
|
||
| /// `XCTestObservation` provides hooks for being notified about progress during a | ||
| /// test run. | ||
| /// - seealso: `XCTestObservationCenter` | ||
| public protocol XCTestObservation: class { | ||
| /// Called just before a test begins executing. | ||
| /// - Parameter testCase: The test case that is about to start. Its `name` | ||
| /// property can be used to identify it. | ||
| func testCaseWillStart(testCase: XCTestCase) | ||
|
|
||
| /// Called when a test failure is reported. | ||
| /// - Parameter testCase: The test case that failed. Its `name` property | ||
| /// can be used to identify it. | ||
| /// - Parameter description: Details about the cause of the test failure. | ||
| /// - Parameter filePath: The path to the source file where the failure | ||
| /// was reported, if available. | ||
| /// - Parameter lineNumber: The line number in the source file where the | ||
| /// failure was reported. | ||
| func testCase(testCase: XCTestCase, didFailWithDescription description: String, inFile filePath: String?, atLine lineNumber: UInt) | ||
|
|
||
| /// Called just after a test finishes executing. | ||
| /// - Parameter testCase: The test case that finished. Its `name` property | ||
| /// can be used to identify it. | ||
| func testCaseDidFinish(testCase: XCTestCase) | ||
| } | ||
|
|
||
| // All `XCTestObservation` methods are optional, so empty default implementations are provided | ||
| public extension XCTestObservation { | ||
| func testCaseWillStart(testCase: XCTestCase) {} | ||
| func testCase(testCase: XCTestCase, didFailWithDescription description: String, inFile filePath: String?, atLine lineNumber: UInt) {} | ||
| func testCaseDidFinish(testCase: XCTestCase) {} | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| // This source file is part of the Swift.org open source project | ||
| // | ||
| // Copyright (c) 2016 Apple Inc. and the Swift project authors | ||
| // Licensed under Apache License v2.0 with Runtime Library Exception | ||
| // | ||
| // See http://swift.org/LICENSE.txt for license information | ||
| // See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors | ||
| // | ||
| // | ||
| // XCTestObservationCenter.swift | ||
| // Notification center for test run progress events. | ||
| // | ||
|
|
||
| /// Provides a registry for objects wishing to be informed about progress | ||
| /// during the course of a test run. Observers must implement the | ||
| /// `XCTestObservation` protocol | ||
| /// - seealso: `XCTestObservation` | ||
| public class XCTestObservationCenter { | ||
|
|
||
| private static var center = XCTestObservationCenter() | ||
| private var observers = Set<ObjectWrapper<XCTestObservation>>() | ||
|
|
||
| /// Registration should be performed on this shared instance | ||
| public class func sharedTestObservationCenter() -> XCTestObservationCenter { | ||
| return center | ||
| } | ||
|
|
||
| /// Register an observer to receive future events during a test run. The order | ||
| /// in which individual observers are notified about events is undefined. | ||
| public func addTestObserver(testObserver: XCTestObservation) { | ||
| observers.insert(testObserver.wrapper) | ||
| } | ||
|
|
||
| /// Remove a previously-registered observer so that it will no longer receive | ||
| /// event callbacks. | ||
| public func removeTestObserver(testObserver: XCTestObservation) { | ||
| observers.remove(testObserver.wrapper) | ||
| } | ||
|
|
||
|
|
||
| internal func testCaseWillStart(testCase: XCTestCase) { | ||
| forEachObserver { $0.testCaseWillStart(testCase) } | ||
| } | ||
|
|
||
| internal func testCase(testCase: XCTestCase, didFailWithDescription description: String, inFile filePath: String?, atLine lineNumber: UInt) { | ||
| forEachObserver { $0.testCase(testCase, didFailWithDescription: description, inFile: filePath, atLine: lineNumber) } | ||
| } | ||
|
|
||
| internal func testCaseDidFinish(testCase: XCTestCase) { | ||
| forEachObserver { $0.testCaseDidFinish(testCase) } | ||
| } | ||
|
|
||
| private func forEachObserver(@noescape body: XCTestObservation -> Void) { | ||
| for observer in observers { | ||
|
There was a problem hiding this comment. You could also follow the function's name 😊 There was a problem hiding this comment. Yeah that's true! I remember considering that but decided to do the traditional |
||
| body(observer.object) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private extension XCTestObservation { | ||
| var wrapper: ObjectWrapper<XCTestObservation> { | ||
| return ObjectWrapper(object: self, objectIdentifier: ObjectIdentifier(self)) | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| // RUN: %{swiftc} %s -o %{built_tests_dir}/Observation | ||
| // RUN: %{built_tests_dir}/Observation > %t || true | ||
| // RUN: %{xctest_checker} %t %s | ||
|
|
||
| #if os(Linux) || os(FreeBSD) | ||
| import XCTest | ||
| #else | ||
| import SwiftXCTest | ||
| #endif | ||
|
|
||
| class Observer: XCTestObservation { | ||
| var startedTestCaseNames = [String]() | ||
| var failureDescriptions = [String]() | ||
| var finishedTestCaseNames = [String]() | ||
|
|
||
| func testCaseWillStart(testCase: XCTestCase) { | ||
| startedTestCaseNames.append(testCase.name) | ||
| } | ||
|
|
||
| func testCase(testCase: XCTestCase, didFailWithDescription description: String, inFile filePath: String?, atLine lineNumber: UInt) { | ||
| failureDescriptions.append(description) | ||
| } | ||
|
|
||
| func testCaseDidFinish(testCase: XCTestCase) { | ||
| finishedTestCaseNames.append(testCase.name) | ||
| } | ||
| } | ||
|
|
||
| let observer = Observer() | ||
|
|
||
| class Observation: XCTestCase { | ||
| static var allTests: [(String, Observation -> () throws -> Void)] { | ||
| return [ | ||
| ("test_one", test_one), | ||
| ("test_two", test_two), | ||
| ("test_three", test_three), | ||
| ] | ||
| } | ||
|
|
||
| // CHECK: Test Case 'Observation.test_one' started. | ||
| // CHECK: Test Case 'Observation.test_one' passed \(\d+\.\d+ seconds\). | ||
| func test_one() { | ||
| XCTAssertEqual(observer.startedTestCaseNames, []) | ||
| XCTAssertEqual(observer.failureDescriptions, []) | ||
| XCTAssertEqual(observer.finishedTestCaseNames, []) | ||
|
|
||
| XCTestObservationCenter.sharedTestObservationCenter().addTestObserver(observer) | ||
| } | ||
|
|
||
| // CHECK: Test Case 'Observation.test_two' started. | ||
| // CHECK: .*/Observation/main.swift:\d+: error: Observation.test_two : failed - fail! | ||
| // CHECK: Test Case 'Observation.test_two' failed \(\d+\.\d+ seconds\). | ||
| func test_two() { | ||
| XCTAssertEqual(observer.startedTestCaseNames, ["Observation.test_two"]) | ||
| XCTAssertEqual(observer.finishedTestCaseNames,["Observation.test_one"]) | ||
|
|
||
| XCTFail("fail!") | ||
| XCTAssertEqual(observer.failureDescriptions, ["failed - fail!"]) | ||
|
|
||
| XCTestObservationCenter.sharedTestObservationCenter().removeTestObserver(observer) | ||
| } | ||
|
|
||
| // CHECK: Test Case 'Observation.test_three' started. | ||
| // CHECK: Test Case 'Observation.test_three' passed \(\d+\.\d+ seconds\). | ||
| func test_three() { | ||
| XCTAssertEqual(observer.startedTestCaseNames, ["Observation.test_two"]) | ||
| XCTAssertEqual(observer.finishedTestCaseNames,["Observation.test_one"]) | ||
| } | ||
| } | ||
|
|
||
| XCTMain([testCase(Observation.allTests)]) | ||
|
|
||
| // CHECK: Executed 3 tests, with 1 failure \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds | ||
| // CHECK: Total executed 3 tests, with 1 failure \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great way of putting it - I wonder whether there'd be value in opening a thread about changing that on Swift evolution. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does seem worth discussing. I was a bit disappointed that I had to jump through such hoops to make
Setwork out in this situation. I'm not sure how soon I'd have time to do a proper write-up though, so please feel free to open a thread if you feel so inclined!