|
| 1 | +# Integration test framework |
| 2 | + |
| 3 | +## Purpose |
| 4 | + |
| 5 | +Extism have many language's SDk and PDK, which is excellent! but we're lack of tests to make sure all SDK and PDK are working expectedly. |
| 6 | + |
| 7 | +for example: we made a change to GO-PDK, tested with GO-SDK, Rust-SDK but breaks JS-SDK. |
| 8 | + |
| 9 | +So we need a integration test framework to test over all SDK/PDK permutations. |
| 10 | + |
| 11 | +## Solution |
| 12 | + |
| 13 | +### Key concepts |
| 14 | + |
| 15 | +#### TestTarget |
| 16 | +Represents the test running environment, |
| 17 | +Includes env(different toolchains), PDK(and/or version), SDK(and/or version). |
| 18 | + |
| 19 | +example: |
| 20 | +```yaml |
| 21 | +env: |
| 22 | + - tiny-go=23.0.1 |
| 23 | +sdk: js-sdk=2.0.0-rc |
| 24 | +pdk: go-pdk=latest |
| 25 | +``` |
| 26 | +
|
| 27 | +env feature is optional. |
| 28 | +
|
| 29 | +#### TestSuite |
| 30 | +Represents a group of TestCases with options. |
| 31 | +
|
| 32 | +options: |
| 33 | + - targets: target matchers, use to skip unimplemented/unsupported target or env |
| 34 | +
|
| 35 | +#### TestCase |
| 36 | +- inside a TestSuite |
| 37 | +- have an expected output to validate |
| 38 | +
|
| 39 | +
|
| 40 | +### Workflow |
| 41 | +
|
| 42 | +1. design TestSuite and TestCases for ALL SDK/PDK |
| 43 | +2. implement TestSuites |
| 44 | + - each PDK/SDK to test should implement the TestSuite code, follow the expected output |
| 45 | + - store in a standardlong repo? |
| 46 | +3. define TestTargets(optional) |
| 47 | + - can define additaional TestTargets to run test suites |
| 48 | +4. trigger runner by new version released or by manual |
| 49 | +
|
| 50 | +
|
| 51 | +### Runflow |
| 52 | +Use your favorite build tools to implement, recomment base on Docker. |
| 53 | +
|
| 54 | +1. scan TestSuite and TestCases to generate inital TestTargets |
| 55 | +2. mix inital TestTargets with additaional TestTargets |
| 56 | +3. prepare environment |
| 57 | + - compile wasm binaries with specified toolchain and PDK |
| 58 | + - prepare SDKs |
| 59 | +4. run all TestTargets |
| 60 | +5. check output is expected and generate test report |
| 61 | +
|
| 62 | +
|
| 63 | +## Considerations |
| 64 | +- need a Runner to run SDK code and PDK binary, also can be used to benchmark |
| 65 | +- better not use assert library to check output, just compare output itself to keep it simple. |
0 commit comments