[SR-710] Generate XCTestCaseProvider entries on Linux

[mxcl]

Previous ID	SR-710
Radar	rdar://problem/32453385
Original Reporter	@mxcl
Type	Improvement

Attachment: Download

Additional Detail from JIRA

Votes	6
Component/s	Package Manager, XCTest
Labels	Improvement
Assignee	None
Priority	Medium

md5: d49557eb297404f7997ff4b6bf1a55f5

Sub-Tasks:

• SR-1613 Require blocks runtime when compiling SourceKit on Linux
• SR-1639 [SourceKit] Add in-process implementations of sourcekitd functions only defined for XPC

is blocked by:

• SR-1676 Build SourceKit on Linux

duplicates:

• SR-32 Find way to make allTests required / audited on OS X

Issue Description:

To test on Linux XCTestCase entries must conform to the protocol XCTestCaseProvider.

This means mostly mac developers probably won't bother making their tests run on Linux, which is not ideal.

XCTestCaseProvider will exist until Swift has reflection support on Linux, so we should try to generate these entries on Linux for now in the PM.

Use the AST to extract XCTestCase classes and their test functions, generate a source, add it to the test module.

Also generate an XCTMain for the full package suite.

This feature depends on me landing testing support (hopefully today).

[modocache]

Understanding that the initial testing support in the swift-package-manager is only intended to support swift-corelibs-xctest, this direction makes me feel a little uneasy.

A lot of the pain around third-party testing frameworks built on top of Xcode (https://github.com/kiwi-bdd/Kiwi/, https://github.com/specta/specta, https://github.com/Quick/Quick) originates from an assumption built into Xcode: that test methods are known at compile time. Because these frameworks dynamically generate test methods using custom functions and methods, Xcode does not support them. As a result, many of the Xcode testing navigator features do not work: no test indicators are displayed next to tests in the line number gutter, for example.

The key flaw in Xcode's testing support is that it does not provide a way for testing frameworks to indicate where tests are located. They cannot report test locations to Xcode, instead Xcode treats XCTestCase subclass methods that begin with "test" and take no arguments as tests, period. So these frameworks simply cannot support Xcode's test navigator features (outside of swizzling Xcode, which is not ideal).

I understand that providing a list of test methods automatically is crucial, and I'm totally on board with that in the short- or medium-term. But I do think this approach will have to be changed eventually in order to be more flexible, so I just thought I'd raise that point for future consideration.

[modocache]

Just to clarify: this is what I mean by " test indicators displayed next to tests in the line number gutter" above:

[mxcl]

Can you clarify your concerns? This work has no direct impact on Xcode.

[modocache]

Ah, sorry I wasn't clear. I used Xcode as an example of how assumptions that test functions will be known at compile time can make things difficult for non-XCTest testing tools. My only concern is that the approach you are proposing (using the AST to extract test functions) may make it harder for the Swift package manager to one day define a protocol that other test frameworks could work with.

[mxcl]

If it makes it harder then we have failed. I see this as strictly a solution for XCTest. Using XCTest should not come with caveats for some platforms.

[modocache]

Thanks for you response

[aciidgh]

should the generator use swiftc's -dump-ast mode to generate AST and then parse the AST to get the test methods names?

[mxcl]

@belkadan is there an AST variety that is considered stable?

[belkadan]

You could do something with SourceKit; talk to @akyrtzi or @nkcsgexi about that. Failing that, no, you would need to write your own tool that walks the AST and prints the information you need.

[akyrtzi]

Note that SourceKit is not supported on linux.

[nkcsgexi]

Would a code completion item help here?

[swift-ci]

Comment by Honza Dvorsky (JIRA)

My attempt: #156

[modocache]

This is a major Swift 3 goal for both SwiftPM and corelibs-xctest, and I'd love to contribute in any way I can. I'm trying to understand the current state of things; here's the work that's been done so far:

• In [WIP] Automatic Linux test manifest generation #156 and Automatic Linux test manifest generation #159 czechboy0 (JIRA User) parses the test files themselves, using string comparison to find func test...() functions.

• In Generate boilerplate code for XCTest on linux using AST #164 @aciidb0mb3r uses -dump-ast to generate a list of tests.

However, both of these approaches have flaws, which @gribozavr sums up well in #159 (comment)

• Parsing Swift with an ad-hoc parsers is doomed to be perpetually buggy and become a maintenance liability.

• Parsing -dump-ast is going to be fragile, or not even work at all, after we remove -dump-ast from production compilers to save code size.

Reflection would be a great tool here, but as @ddunbar mentions, it doesn't seem like those APIs are available in Swift yet. I've searched the mailing lists, but all I could find was this email that stated it was "one of our stretch goals for Swift 3".

@gribozavr and others in this thread have advocated instead to move the test scanning code from SourceKit into libIDE, and provide a documented and supported driver option in the Swift compiler to scan for tests. @ddunbar and others (myself included) have expressed misgivings in requiring test methods to be known at compile time, but we may need to punt on that discussion until after we achieve our Swift 3 goal of supporting XCTest.

SourceKit already has code to detect whether an instance method defines a test in XCTest: see SourceKit::FuncDeclEntityInfo.IsTestCandidate. So why not use SourceKit? SourceKit is meant to be an asynchronous wrapper for libIDE, and it implements that asynchronicity using XPC, which only works on OS X.

So if we move something like IsTestCandidate to libIDE, we can just use that, right? Well, not exactly, since that's a C++ library that we can't use directly from Swift (both SwiftPM and corelibs-xctest are written in Swift).

Seems to me there are several potential solutions here:

1. Port SourceKit to Linux, using a different form of IPC on platforms where XPC is not available.

2. Move business logic like IsTestCandidate to libIDE and add a C header (like libclang or sourcekitd). SwiftPM/XCTest will link against libIDE and use the C header.

3. Move business logic like IsTestCandidate to libIDE and add a swiftc option to interface with its functionality. SwiftPM/XCTest will use swiftc.

I think #2 is the best option. It's less work than both #1 and #3. Logic like IsTestCandidate belongs in libIDE anyway--SourceKit should stick to XPC and asynchronous communication with libIDE.

Thoughts? Please comment on this issue if you have feedback or would be interested in taking this work on!

[aciidgh]

@modocache I can handle the swiftpm/xctest part of this

[drewcrawford]

I'm also invested in finding a solution here. Thanks to @modocache for an excellent summary.

Unfortunately, I have a concern completely out of left field. I would rather wish it away, but I'm afraid I've tried that, and it is still bothering me.

Existentialist crisis: what is a test? Pretend for a moment we have solved where to house IsTestCandidate. What is its definition? What does the function do?

For XCTest the answer is obvious: a test is a function that subclasses XCTestCase and starts with test. But is that the definition of a "test candidate"? It sounds more to me like an "*xc*test candidate". At best, we have commingled an internal implementation detail of XCTest with an internal implementation detail of libIDE, SourceKit, swiftc, or wherever this lands.

Meanwhile in another part of town, owensd (JIRA User) has been playing around with syntaxes for writing tests inline. Now, are they "tests" in the XCTest sense? Obviously not. But are they tests? I think they clearly are, and so if there is some compiler-level oracle that classifies functions as tests, it should classify these, in which case the oracle will no longer be useful for XCTest.

Now stepping way out into left field: does the problem begin and end with simply deciding if something is a test or not? e.g., what if I want to skip a test, or expect it to fail, or expect it to fail on Linux, or don't run it in parallel, or any number of other circumstances that actually arise in e.g. the Swift compiler. Now I have to create a bunch of metadata for each test, populate it in a separate file, so we know to skip it on Wednesdays or whatever the business logic requires.

But we are here because we have metadata in separate files now, and we think it is unmanageable. So right after we solve this we will back here figuring out where to house isParallelTestCandidate, isLinuxTestCandidate, etc.

My suggestion is that the interface we need is very different than isTestCandidate. The question of "what is a test?" has no universal answer and should be implemented inside XCTest, where the answer is clear and it is very appropriate to view the world through XCTest lens. And on the compiler side, we need tooling to allow XCTest to own its definition, so that it can evolve, change, experiment without compiler precoordination, and for that matter so can other ideas about test frameworks.

I believe we have a few options here:

1. A real machine-parseable source representation, as distinct from -dump-ast which is internal. Behind this door we would specifically craft a "file format" and consider it API. It could contain, initially, just the parts of the parse XCTest needs to find tests, but could be extended as needed to support other uses.
2. An oracle that can answer arbitrary queries about Swift programs. Behind this door, XCTest would ask the oracle questions like: "Find me all subclasses of XCTestCase." "Find instance methods of class Foo that start with test". And based on this information XCTest will classify things as tests or not. In this scenario we have another choice: whether our oracle should be static (e.g. in libIDE or similar) or dynamic (e.g. in the runtime / reflection). Figuring out where to store metadata in the dynamic case is a bit sticky, but it may be possible. But in any case the distinction is that, while current libIDE proposals assume a single API call, I think the surface area is larger.

In summary, I really appreciate all the energy and effort going into solving this. My concern is making sure we actually do solve it, so we are not back here again a few months from now.