Proposal: Unified render testing API for Kotlin and Swift

[zach-klippenstein]

First draft of this proposal came out of a discussion between @zach-klippenstein and @bencochran.

Background

Swift and Kotlin built completely separate testing APIs. Swift has two APIs: one for testing actions, and one for testing rendering (although it also ends up testing actions). Kotlin also has two APIs, but split on a different axis: one for integration testing whole workflow trees, and one for testing single render passes. The kotlin one has some rough edges, namely #489 and #499, and there's a lot of confusion about how to use it.

#687 is an attempt to build a Kotlin testing API that more closely matched the Swift render testing API. Swift contributors raised some concerns over their API as well though, and so we held off on merging that until we could talk through all the issues and come up with a better solution that met all the requirements.

This proposal is only concerned with the API for unit testing single render passes.

Concerns with existing APIs

Kotlin

1. Requires passing in fake child workflows and workers. Since most workflows inject their children, this is required anyway, but the current API requires child workflows to actually return a rendering. Fakes that are required to have some behavior, but not all the behavior (they can't run any grandchildren or anything) is confusing and feels half-baked. Moreover, it's not clear how to actually create these mocks (do you use MockChildWorkflow or mockito mocks?).
2. Because child renderings must be returned from the fake workflows, it's not easy to create tests that verify a wide variety of child outputs - the parent workflow itself has to be setup with different fake children every time.

Swift

1. ExpectedOutput is actually an assertion on the result of the render pass. It's confusing because the naming and API looks more similar to the API for setting up expected child workflows/workers.
2. Render tester object is actually stateful, and invoking render event handlers, for example, will actually mutate the state inside of the tester object instead of being pure function calls.
3. RenderTester.render is a fluent-style API – it returns self so you can chain tests for multiple consecutive render passes. This encourages bloated tests, instead of a 1-to-1 unit test per render pass relation.
4. Because the only way to assert the result of children and events is by asserting on their output (with ExpectedOutput), render tests are actually also testing WorkflowActions. Actions should be tested separately.
5. Passing all the expectations and assertions into one giant function call, and requiring manually creating types like ExpectedWorkflow and ExpectedWorker is clunky and exposes too many types that should probably remain internal.

Proposed enhancements

Kotlin will get a testing API much more similar to Swifts, using #687 as a starting point. The high-level API usage will look as follows. The testing API is divided into multiple stages.

1. Stage 1: Setup
   1. Specify input state (and props, for kotlin) for the render call.
   2. (Optional) Specify child workflow and worker expectations. Any children not explicitly "expected" will be considered test failures.
      • Expectations specify how to match the actual child being run (the type + key for workflows, custom matcher for workers).
      • For workflows, they also specify an assertion function to assert that the parent passes the right props to the child, and the rendering that the child should "return".
      • Between all child workflows and workers, one may specify an output. It is an error if more than one child is given an output. If an output is specified, the test will be allowed to assert on the action that is invoked to handle the output.
2. Stage 2: Render
   1. The workflow-under-test's render method is invoked, and the returned rendering is given to the test. The test can make assertions on the rendering, as well as invoke an event handler.
   2. If a rendering event is invoked, the corresponding action to handle the event is stored by the testing framework to be validated later. In this case, a few constraints apply:
      • Only one event may be invoked per test.
      • No child workflows or workers may have had outputs specified.
   • Note that this stage does not actually execute any WorkflowActions, and is effectively a pure function call.
3. Stage 3: Action verification (Optional)
   If an output was specified or a rendering event was invoked, the test will need to assert that the correct action handled the event. There are two ways to verify actions:
   • If the action is a concrete type, such as actions that are defined as sealed classes/enums-with-associated-values, the action instance itself can simply be compared. This style allows the test to just perform an equality check on the WorkflowAction instance.
   • If the action was anonymous, the test has nothing to compare it to. In this case, the anonymous action will be executed and the resulting (state, output) pair will be given to the test to assert upon.

API details

The last stage will not allow fluent calls. If you want to test the next render pass or resulting concrete action, write another test.

Whether the optional phases of setup are passed into the same function as the required ones (as is currently the case in the Kotlin PR and the existing Swift API), or specified in more of a fluent builder style, is an open question. It's also mostly a question of style, so it should be feasible to sketch both out and see what is more convenient in practice. See the Swift concerns above about exposing extra types.

cc @rjrjr @AquaGeek

[rjrjr]

• No proposal to make parallel updates to the Swift api?
• Should we give Swift integration test support like Kotlin's?

[zach-klippenstein]

This proposal is for both Kotlin and Swift, as the title indicates.
This proposal is only about render testing, swift integration testing should be discussed in another issue.

[rjrjr]

Cool. Phrasing of Kotlin will get a testing API much more similar to Swifts, using #687 as a starting point confused me, made it sound like only Kotlin work was actually planned.

[zach-klippenstein]

Fair enough, I will try to get to Swift after Kotlin, maybe pair with @bencochran.