[SR-907] Add convenience XCTestExpectation constructors for NSNotification

[modocache]

Previous ID	SR-907
Radar	None
Original Reporter	@modocache
Type	Improvement
Status	Closed
Resolution	Done

Attachment: Download

Additional Detail from JIRA

Votes	0
Component/s	XCTest
Labels	Improvement, StarterBug
Assignee	apps.yoon (JIRA)
Priority	Medium

md5: 2bc0310ca30f22236b57ae9be6504fd5

Issue Description:

Apple XCTest offers several convenient methods for constructing XCTestExpectations:

@interface XCTestCase : XCTest
// ...
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath expectedValue:(nullable id)expectedValue;
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath handler:(nullable XCKeyValueObservingExpectationHandler)handler;
- (XCTestExpectation *)expectationForNotification:(NSString *)notificationName object:(nullable id)objectToObserve handler:(nullable XCNotificationExpectationHandler)handler;
- (XCTestExpectation *)expectationForPredicate:(NSPredicate *)predicate evaluatedWithObject:(id)object handler:(nullable XCPredicateExpectationHandler)handler;
// ...
@end

Key-value observing doesn't exist in native Swift, but NSNotification and NSPredicate do (thanks to swift-corelibs-foundation). Add the following convenience constructors:

1. expectationForNotification:object:handler:
2. expectationForPredicate:evaluatedWithObject:handler:

Also, note that you'll need to add the following closure typealiases as well:

typedef BOOL (^XCNotificationExpectationHandler)(NSNotification *notification);
typedef BOOL (^XCPredicateExpectationHandler)();

[swift-ci]

Comment by Kyle Yoon (JIRA)

@modocache I would like to pick this up. First time contributing here. I've been looking at the pull requests for swift-corelibs-xctest, but to be frank, I'm still a bit lost on how to go about this. Are there any examples or guidelines that could help me get started?

[modocache]

Hello apps.yoon (JIRA User)!

1. First of all, make sure you can build the project: follow the instructions in the README to do so. Keep in mind that on OS X, you'll need a Swift 3 .xctoolchain, which you'll need to build yourself (see utils/build-toolchain in the main Swift repo). It may be easier to work on Linux instead: you can simply run the Swift build script: utils/build-script --xctest.

2. Next, make sure you can run the tests, and that they all pass. There are instructions on how to run the tests in the README. You can also use the Swift build script: utils/build-script --xctest --test – --skip-test-swift.

3. Next, add a test to the Tests/Functional directory. fe76924 adds a test. Check that commit out to see an example. Your test should try creating an expectation using XCTestCase.expectationForNotification(notification:object:handler: ). Run the tests to confirm that this test fails--after all, you haven't added the XCTestCase.expectationForNotification(notification:object:handler: ) method yet!

4. Define a `public` method XCTestCase.expectationForNotification(notification:object:handler: ), and make sure your test can call it.

...then implement the method to do what it does in Apple XCTest! You can confirm how Apple XCTest behaves by starting a new Xcode project with unit tests, then using the XCTestCase.expectationForNotification(notification:object:handler: ) method in there.

Let me know if you have any specific questions along the way!

[swift-ci]

Comment by Kyle Yoon (JIRA)

Oh wow! Thank you for the great instructions. For one, I was wondering how to get the latest Swift 3 xctoolchain because I didn't see it in swift.org downloads. I'll probably have some more questions but this is definitely going to get me started. I'll go ahead and assign this to myself then?

[modocache]

Sure thing! Let me know how things go.

[modocache]

By the way, here's a Swift 3 .xctoolchain I built yesterday, in case you still need one: https://mega.nz/#!iB8TSbbJ!pCmtcgAoYsljwBiQALy7_s9qwXR2egQJ27ReBlq5hGo

[swift-ci]

Comment by Kyle Yoon (JIRA)

Thanks for the .xctoolchain! I'm having trouble making the tests pass for step 2. Since the toolchain is from 3/16, I rolled back the foundation repo to the last commit before then. I then launch Xcode with the toolchain, build SwiftXCTest, and run SwiftXCTestFunctionalTests. This all goes successfully. But when I run the command xcodebuild -workspace XCTest.xcworkspace -scheme SwiftXCTestFunctionalTests, the build fails.

The following build commands failed:
CompileSwift normal x86_64 /Users/kyleyoon/Development/swift-corelibs-foundation/Foundation/NSArray.swift
CompileSwift normal x86_64 /Users/kyleyoon/Development/swift-corelibs-foundation/Foundation/NSEnergyFormatter.swift
CompileSwift normal x86_64 /Users/kyleyoon/Development/swift-corelibs-foundation/Foundation/NSURLError.swift
CompileSwift normal x86_64 /Users/kyleyoon/Development/swift-corelibs-foundation/Foundation/NSPersonNameComponentsFormatter.swift
CompileSwiftSources normal x86_64 com.apple.xcode.tools.swift.compiler
(5 failures)

Is there something I'm missing here?

[swift-ci]

Comment by Kyle Yoon (JIRA)

Hmmm my guess is that I need to run the command with the Swift 3 toolchain...

[modocache]

Oops, sorry Kyle. I tried responding to your comment via email but it looks like JIRA doesn't support that feature.

Building and running in Xcode is sufficient. xcodebuild on the command line won't work unless your Xcode uses the 3/16 toolchain by default. I think this might be possible in Xcode 7.3, I'm not sure.

For now, stick to building and running in the Xcode app--that's all you'll need.

I'm working on adding more detailed instructions to the README that should prevent confusion in the future--stay tuned!

[swift-ci]

Comment by Kyle Yoon (JIRA)

No worries!

I've made some progress - wrote my first implementation of the {XCTestCase.expectationForNotification(notification:object:handler: )} and some tests in the {main.swift} file of the Expectations directory. However, I've run into another issue regarding tests that are expected to fail.

Here's one of the tests I wrote:

// CHECK: Test Case 'ExpectationsTestCase.test_observeNotificationWithIncorrectName_fails' started.
// CHECK: */Tests/Functional/Asynchronous/Expectations/main.swift:125: error: ExpectationsTestCase.test_observeNotificationWithIncorrectName_fails : Asynchronous wait failed - Exceeded timeout of 1.0 seconds, with unfulfilled expectations: notification
// CHECK: Test Case 'ExpectationsTestCase.test_observeNotificationWithIncorrectName_fails' failed \(\d+\.\d+ seconds\).
    func test_observeNotificationWithIncorrectObject_fails() {
        let notificationName = "notificationWithIncorrectObjectTest"
        let dummyObject = NSObject()
        let anotherDummyObject = NSObject()
        expectationForNotification(notificationName, object: dummyObject, handler:nil)
        NSNotificationCenter.defaultCenter().postNotificationName(notificationName, object: anotherDummyObject)
        waitForExpectationsWithTimeout(0.1, handler: nil)
    }

I've been able to fix up my tests when Xcode gives me errors pointing to specific CHECKs, but right now, it's just telling me the build failed. I'm having trouble without breakpoints, are there any other debugging methods that you suggest I try out?

[modocache]

Try using the ⌘8 shortcut to show the report navigator in Xcode, then click on the build that failed:

Scroll to the bottom of the build log, and you should be able to see detailed failure messages.

@briancroom has done some awesome work in making these failure messages more prominently displayed in Xcode: #72 Based on your experience, though, I imagine even more improvements could be made here! 🙂

[swift-ci]

Comment by Kyle Yoon (JIRA)

I just updated to XCode 7.3, would that make xcrun launch-with-toolchain not work anymore?

[modocache]

I believe Xcode 7.3 has a preference pane that allows you to choose an .xctoolchain. Search for "Xcode 7.3" on this page: https://swift.org/download/#using-downloads

You should use that instead of xcrun launch-with-toolchain.

[swift-ci]

Comment by Kyle Yoon (JIRA)

I couldnt' find the Toolchains tab in the Components tab, but turns out you have to have the Toolchains directory in the default Library/Developer/ path. I've been launching Xcode with the custom toolchain from another directory previous to the update. Oops!

[swift-ci]

Comment by Kyle Yoon (JIRA)

@modocache, I'm having trouble using NSPredicate in my tests because the initializers are NSUnimplemented(). The error message when running tests with NSPredicate is "Init(format:argumentArray:) is not yet implemented: file /Users/kyleyoon/Development/swift-corelibs-foundation/Foundation/NSPredicate.swift, line 36". Is this a blocker for adding expectationForPredicate:evaluatedWithObject:handler:?

[modocache]

Ah yes, it seems like NSPredicate is completely unimplemented in swift-corelibs-foundation: https://github.com/apple/swift-corelibs-foundation/blob/338f4bf3a89c75a0420b49f5701466e106af02b5/Foundation/NSPredicate.swift

Is there any discussion on the swift-corelibs-dev mailing list, or any issues related to NSPredicate on Swift JIRA? We'll need NSPredicate implemented in Swift in order to meet our Swift 3 goals for XCTest. If no one's working on it, we should create a task to track progress--and potentially begin working on it ourselves!

As for this task, I think it'll be easier to begin by working on NSNotification-related expectations.

Thanks for discovering this, apps.yoon (JIRA User)!! I know this seems like a setback, but I think it's an important discovery, and I'm glad you brought it up.